2024-05-08T00:19:03.307 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-08T00:19:03.308 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_dis@teuthology --verbose --archive /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630 --name dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi --description rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} -- /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630/orig.config.yaml 2024-05-08T00:19:03.353 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T00:19:03.446 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630 branch: wip-dis-testing-squid description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} email: idryomov@gmail.com first_in_suite: false job_id: '7696630' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-dis-testing-squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: e0c1691aade07c0b78ccff9531f65d3c80e3e220 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: e0c1691aade07c0b78ccff9531f65d3c80e3e220 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-squid sha1: 9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 5306 sha1: e0c1691aade07c0b78ccff9531f65d3c80e3e220 sleep_before_teardown: 0 subset: 21/99 suite: rbd suite_branch: wip-dis-testing-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: 9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNse/AFoy1yDcGyY+MS/1v1sY6rGKOdpmwkIA7spCDsdeG/eq/LWdPvBzKpmx6d2QajiJpoDqcDNFeS5zRkvOOI= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJY4Atyn5pq0iMFwyGZY2SznIWkQuGBB2prrTIQN9ZryfhoD6Ci81KzAnIOQdzmtaHC2/SgzepfCshe0z9Vzcbc= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' RBD_MIRROR_MIN_COMPAT_CLIENT: octopus RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-07_23:02:08 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-08T00:19:03.446 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa; will attempt to use it 2024-05-08T00:19:03.447 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa/tasks 2024-05-08T00:19:03.447 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-08T00:19:03.448 INFO:teuthology.task.internal:Checking packages... 2024-05-08T00:19:03.472 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e0c1691aade07c0b78ccff9531f65d3c80e3e220' 2024-05-08T00:19:03.472 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-08T00:19:03.472 INFO:teuthology.packaging:ref: None 2024-05-08T00:19:03.472 INFO:teuthology.packaging:tag: None 2024-05-08T00:19:03.472 INFO:teuthology.packaging:branch: wip-dis-testing-squid 2024-05-08T00:19:03.472 INFO:teuthology.packaging:sha1: e0c1691aade07c0b78ccff9531f65d3c80e3e220 2024-05-08T00:19:03.472 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-dis-testing-squid 2024-05-08T00:19:03.689 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2529.ge0c1691a 2024-05-08T00:19:03.690 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-08T00:19:03.698 INFO:teuthology.task.internal:no buildpackages task found 2024-05-08T00:19:03.698 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-08T00:19:03.745 INFO:teuthology.task.internal:Saving configuration 2024-05-08T00:19:03.762 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-08T00:19:03.796 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-08T00:19:03.820 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 00:13:58.168236', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNse/AFoy1yDcGyY+MS/1v1sY6rGKOdpmwkIA7spCDsdeG/eq/LWdPvBzKpmx6d2QajiJpoDqcDNFeS5zRkvOOI='} 2024-05-08T00:19:03.842 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 00:13:58.166371', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJY4Atyn5pq0iMFwyGZY2SznIWkQuGBB2prrTIQN9ZryfhoD6Ci81KzAnIOQdzmtaHC2/SgzepfCshe0z9Vzcbc='} 2024-05-08T00:19:03.842 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-08T00:19:03.847 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-05-08T00:19:03.847 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-05-08T00:19:03.847 INFO:teuthology.run_tasks:Running task console_log... 2024-05-08T00:19:03.963 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb1944490d0>, signals=[15]) 2024-05-08T00:19:03.964 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-08T00:19:03.970 INFO:teuthology.task.internal:Opening connections... 2024-05-08T00:19:03.970 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T00:19:03.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:19:04.050 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T00:19:04.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:19:04.125 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-08T00:19:04.132 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-05-08T00:19:04.150 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-05-08T00:19:04.150 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-05-08T00:19:04.207 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-05-08T00:19:04.207 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-05-08T00:19:04.207 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-05-08T00:19:04.207 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-05-08T00:19:04.207 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-05-08T00:19:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-05-08T00:19:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-08T00:19:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-05-08T00:19:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-05-08T00:19:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-08T00:19:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-05-08T00:19:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-08T00:19:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-08T00:19:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-08T00:19:04.209 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-05-08T00:19:04.236 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-05-08T00:19:04.254 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-05-08T00:19:04.254 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-05-08T00:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-05-08T00:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-05-08T00:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-05-08T00:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-08T00:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-08T00:19:04.312 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-05-08T00:19:04.336 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-08T00:19:04.346 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-08T00:19:04.396 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-08T00:19:04.396 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-05-08T00:19:04.399 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-05-08T00:19:04.415 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-08T00:19:04.445 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-08T00:19:04.446 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-05-08T00:19:04.456 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-05-08T00:19:04.489 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-08T00:19:04.542 INFO:teuthology.run_tasks:Running task kernel... 2024-05-08T00:19:04.556 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-05-08T00:19:04.556 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-05-08T00:19:04.556 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-08T00:19:04.557 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-08T00:19:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T00:19:04.573 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-05-08T00:19:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T00:19:04.574 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-05-08T00:19:04.631 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-437.el9.x86_64 2024-05-08T00:19:04.631 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-437.el9.x86_64 2024-05-08T00:19:04.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-05-08T00:19:04.632 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-437.el9.x86_64 2024-05-08T00:19:04.632 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-437.el9.x86_64 2024-05-08T00:19:04.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-05-08T00:19:06.841 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-05-08T00:19:08.102 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.1 MB 00:01 2024-05-08T00:19:08.817 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-05-08T00:19:09.819 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-05-08T00:19:14.085 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.2 MB 00:01 2024-05-08T00:19:15.328 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-05-08T00:19:15.493 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 91 kB/s | 16 kB 00:00 2024-05-08T00:19:17.029 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-05-08T00:19:17.664 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-05-08T00:19:17.855 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 31 MB/s | 21 MB 00:00 2024-05-08T00:19:22.970 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-08T00:19:22.988 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-08T00:19:24.969 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-08T00:19:24.970 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-08T00:19:25.054 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:19:25.059 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-08T00:19:25.059 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-08T00:19:25.063 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:19:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-05-08T00:19:25.063 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-05-08T00:19:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:19:25.065 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 78 M 2024-05-08T00:19:25.065 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 123 M 2024-05-08T00:19:25.065 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:19:25.151 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:19:25.166 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:19:25.166 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-05-08T00:19:25.166 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:19:25.166 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:19:25.166 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-08T00:19:25.166 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:19:25.166 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 78 M 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 123 M 2024-05-08T00:19:25.167 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:19:25.348 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 2.0 MB/s | 484 kB 00:00 2024-05-08T00:19:26.204 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 549 kB/s | 484 kB 00:00 2024-05-08T00:19:26.871 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 10 MB/s | 15 MB 00:01 2024-05-08T00:19:27.638 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 15 MB/s | 34 MB 00:02 2024-05-08T00:19:27.931 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 5.5 MB/s | 15 MB 00:02 2024-05-08T00:19:28.571 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 12 MB/s | 28 MB 00:02 2024-05-08T00:19:28.571 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:19:28.572 INFO:teuthology.orchestra.run.smithi064.stdout:Total 23 MB/s | 78 MB 00:03 2024-05-08T00:19:29.278 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:19:29.691 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:19:29.691 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:19:30.622 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:19:30.622 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:19:31.031 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 5.7 MB/s | 34 MB 00:05 2024-05-08T00:19:31.765 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 4.4 MB/s | 28 MB 00:06 2024-05-08T00:19:31.765 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:19:31.765 INFO:teuthology.orchestra.run.smithi116.stdout:Total 12 MB/s | 78 MB 00:06 2024-05-08T00:19:32.442 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:19:32.779 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:19:32.779 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:19:33.176 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:19:33.707 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:19:33.707 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:19:33.945 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-08T00:19:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-08T00:19:36.132 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:19:36.808 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-08T00:19:36.840 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-08T00:19:36.890 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-08T00:19:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-08T00:19:39.569 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-08T00:19:39.716 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-08T00:19:42.060 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-08T00:19:42.100 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:19:44.459 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-08T00:19:44.585 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:19:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:19:49.325 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:20:17.997 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:20:18.037 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:20:18.448 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:20:18.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-08T00:20:18.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-08T00:20:18.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-08T00:20:18.853 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:20:18.891 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:20:19.204 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:20:19.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-08T00:20:19.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-08T00:20:19.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-08T00:20:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:20:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:20:19.258 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:20:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-08T00:20:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-08T00:20:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-08T00:20:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-08T00:20:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:20:19.258 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:20:19.596 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T00:20:19.667 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-08T00:20:19.667 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-08T00:20:19.667 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-08T00:20:19.667 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-08T00:20:19.667 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-08T00:20:19.668 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-05-08T00:20:19.668 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-05-08T00:20:19.668 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T00:20:19.719 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-08T00:20:19.720 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-08T00:20:19.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-05-08T00:20:19.760 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-05-08T00:20:19.760 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-05-08T00:20:19.760 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-05-08T00:20:19.760 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-05-08T00:20:19.760 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-05-08T00:20:19.761 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-05-08T00:20:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:20:19.762 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-08T00:20:19.762 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-08T00:20:19.762 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-08T00:20:19.762 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-05-08T00:20:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:20:19.762 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-05-08T00:20:19.763 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-08T00:20:19.763 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-444.el9.x86_64 2024-05-08T00:20:19.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-08T00:20:20.000 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-08T00:20:20.000 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:20:20.000 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:20:20.000 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-08T00:20:20.000 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-08T00:20:20.000 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-08T00:20:20.000 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-08T00:20:20.000 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:20:20.000 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:20:20.342 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T00:20:20.353 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-05-08T00:20:20.417 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-08T00:20:20.418 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-08T00:20:20.418 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-08T00:20:20.418 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-08T00:20:20.418 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-08T00:20:20.418 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-05-08T00:20:20.418 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-05-08T00:20:20.418 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-05-08T00:20:20.468 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-08T00:20:20.468 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-08T00:20:20.468 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-05-08T00:20:20.499 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-05-08T00:20:20.499 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-05-08T00:20:20.499 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-05-08T00:20:20.499 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-05-08T00:20:20.499 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:20:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-08T00:20:20.501 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-08T00:20:20.501 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-08T00:20:20.501 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-05-08T00:20:20.501 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:20:20.501 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-05-08T00:20:20.502 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-08T00:20:20.502 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-444.el9.x86_64 2024-05-08T00:20:20.502 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-08T00:20:21.084 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-05-08T00:20:21.689 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-08T00:20:21.725 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-05-08T00:20:21.727 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-05-08T00:20:21.749 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.fWq9Idqrs7 2024-05-08T00:20:21.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fWq9Idqrs7 2024-05-08T00:20:21.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.fWq9Idqrs7 2024-05-08T00:20:21.954 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.fWq9Idqrs7 is 6KB 2024-05-08T00:20:21.974 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.fWq9Idqrs7 2024-05-08T00:20:21.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-05-08T00:20:22.050 INFO:teuthology.orchestra.run.smithi064.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-08T00:20:22.050 INFO:teuthology.orchestra.run.smithi064.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-08T00:20:22.050 INFO:teuthology.orchestra.run.smithi064.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-08T00:20:22.050 INFO:teuthology.orchestra.run.smithi064.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-08T00:20:22.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-08T00:20:22.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-05-08T00:20:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-08T00:20:22.470 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-05-08T00:20:22.471 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-05-08T00:20:22.488 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.qQSLs903KY 2024-05-08T00:20:22.488 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qQSLs903KY 2024-05-08T00:20:22.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.qQSLs903KY 2024-05-08T00:20:22.826 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.qQSLs903KY is 6KB 2024-05-08T00:20:22.874 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.qQSLs903KY 2024-05-08T00:20:22.889 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-05-08T00:20:22.953 INFO:teuthology.orchestra.run.smithi116.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-08T00:20:22.953 INFO:teuthology.orchestra.run.smithi116.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-08T00:20:22.953 INFO:teuthology.orchestra.run.smithi116.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-08T00:20:22.953 INFO:teuthology.orchestra.run.smithi116.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-08T00:20:22.955 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-08T00:20:23.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-05-08T00:20:52.234 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-05-08T00:20:52.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-05-08T00:20:52.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:20:53.091 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-05-08T00:20:53.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-05-08T00:20:53.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:21:10.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-05-08T00:21:11.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-05-08T00:21:19.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-05-08T00:21:19.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:21:20.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-05-08T00:21:20.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:21:22.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-05-08T00:21:23.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-05-08T00:21:34.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-05-08T00:21:34.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:21:35.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-05-08T00:21:35.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:21:53.059 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-05-08T00:22:07.533 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-05-08T00:22:07.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-05-08T00:22:07.897 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-444.el9.x86_64"... 2024-05-08T00:22:07.897 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-05-08T00:22:07.911 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-444.el9.x86_64 2024-05-08T00:22:07.911 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-08T00:22:07.912 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-08T00:22:07.912 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-08T00:22:08.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-05-08T00:22:08.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:22:08.592 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-05-08T00:22:08.912 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-05-08T00:22:08.912 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-08T00:22:08.999 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-05-08T00:22:09.025 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-05-08T00:22:09.026 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-444.el9.x86_64"... 2024-05-08T00:22:09.026 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-05-08T00:22:09.038 DEBUG:teuthology.parallel:result is None 2024-05-08T00:22:09.043 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-444.el9.x86_64 2024-05-08T00:22:09.043 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-08T00:22:09.043 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-08T00:22:09.043 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-08T00:22:10.044 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-05-08T00:22:10.045 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-08T00:22:10.117 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-05-08T00:22:10.150 DEBUG:teuthology.parallel:result is None 2024-05-08T00:22:10.150 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-08T00:22:10.158 INFO:teuthology.task.internal:Creating test directory... 2024-05-08T00:22:10.158 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-08T00:22:10.161 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-08T00:22:10.176 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-08T00:22:10.206 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-08T00:22:10.256 INFO:teuthology.task.internal:Creating archive directory... 2024-05-08T00:22:10.257 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-08T00:22:10.259 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-08T00:22:10.373 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-08T00:22:10.380 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-08T00:22:10.380 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-08T00:22:10.382 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-08T00:22:10.410 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T00:22:10.421 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T00:22:10.484 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T00:22:10.494 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T00:22:10.496 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-08T00:22:10.505 INFO:teuthology.task.internal:Configuring sudo... 2024-05-08T00:22:10.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-08T00:22:10.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-08T00:22:10.532 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-08T00:22:10.568 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-08T00:22:10.569 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-08T00:22:10.575 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-08T00:22:10.590 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T00:22:10.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T00:22:10.720 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T00:22:10.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T00:22:10.845 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:22:10.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-08T00:22:10.912 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T00:22:10.954 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T00:22:11.000 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T00:22:11.083 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T00:22:11.122 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:22:11.122 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-08T00:22:11.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-05-08T00:22:11.188 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-05-08T00:22:11.231 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T00:22:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T00:22:11.718 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-08T00:22:11.726 INFO:teuthology.task.internal:Starting timer... 2024-05-08T00:22:11.726 INFO:teuthology.run_tasks:Running task pcp... 2024-05-08T00:22:11.752 INFO:teuthology.run_tasks:Running task selinux... 2024-05-08T00:22:11.801 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-08T00:22:11.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-05-08T00:22:11.856 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-05-08T00:22:11.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-05-08T00:22:11.913 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-05-08T00:22:11.915 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-08T00:22:11.915 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-05-08T00:22:11.936 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-05-08T00:22:11.936 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-05-08T00:22:11.958 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-05-08T00:22:11.958 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi116.front.sepia.ceph.com': 'permissive'} 2024-05-08T00:22:11.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-08T00:22:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T00:22:12.002 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-08T00:22:12.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T00:22:12.028 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-08T00:22:12.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-05-08T00:22:12.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-05-08T00:22:12.094 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-08T00:22:12.104 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-08T00:22:12.109 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-08T00:22:12.238 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-08T00:22:12.257 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-08T00:22:12.259 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi116.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-08T00:26:14.296 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com')] 2024-05-08T00:26:14.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-05-08T00:26:14.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:26:14.379 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-05-08T00:26:14.463 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-05-08T00:26:14.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-05-08T00:26:14.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T00:26:14.539 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-05-08T00:26:14.619 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-05-08T00:26:14.619 INFO:teuthology.run_tasks:Running task clock... 2024-05-08T00:26:14.631 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-08T00:26:14.631 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-08T00:26:14.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T00:26:14.633 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-08T00:26:14.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T00:26:14.663 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-08T00:26:14.678 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-08T00:26:14.689 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-08T00:26:14.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-05-08T00:26:14.704 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-08T00:26:14.715 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-05-08T00:26:14.727 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-05-08T00:26:14.730 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-08T00:26:14.738 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-05-08T00:26:14.745 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-08T00:26:14.752 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-08T00:26:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-08T00:26:14.806 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-05-08T00:26:14.809 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T00:26:14.809 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-05-08T00:26:14.809 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T00:26:14.809 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T00:26:14.809 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T00:26:14.809 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T00:26:14.831 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-05-08T00:26:14.835 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T00:26:14.835 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-05-08T00:26:14.835 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T00:26:14.835 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T00:26:14.835 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T00:26:14.835 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T00:26:14.836 INFO:teuthology.run_tasks:Running task install... 2024-05-08T00:26:14.844 DEBUG:teuthology.task.install:project ceph 2024-05-08T00:26:14.844 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}} 2024-05-08T00:26:14.845 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'} 2024-05-08T00:26:14.845 INFO:teuthology.task.install:Using flavor: default 2024-05-08T00:26:14.850 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-08T00:26:14.850 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-05-08T00:26:14.850 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220', 'tag': None, 'wait_for_package': False} 2024-05-08T00:26:14.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e0c1691aade07c0b78ccff9531f65d3c80e3e220 2024-05-08T00:26:14.853 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220', 'tag': None, 'wait_for_package': False} 2024-05-08T00:26:14.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e0c1691aade07c0b78ccff9531f65d3c80e3e220 2024-05-08T00:26:15.000 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/centos/9/flavors/default/ 2024-05-08T00:26:15.001 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2529.ge0c1691a 2024-05-08T00:26:15.004 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/centos/9/flavors/default/ 2024-05-08T00:26:15.004 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2529.ge0c1691a 2024-05-08T00:26:15.172 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-08T00:26:15.172 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:26:15.172 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-08T00:26:15.197 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-05-08T00:26:15.198 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e0c1691aade07c0b78ccff9531f65d3c80e3e220/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-08T00:26:15.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-08T00:26:15.338 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-08T00:26:15.402 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-05-08T00:26:15.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-05-08T00:26:15.421 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-dis-testing-squid/e0c1691aade07c0b78ccff9531f65d3c80e3e220/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-08T00:26:15.421 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:26:15.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-08T00:26:15.450 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-05-08T00:26:15.451 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e0c1691aade07c0b78ccff9531f65d3c80e3e220/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-08T00:26:15.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-08T00:26:15.597 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-08T00:26:15.650 INFO:teuthology.orchestra.run.smithi116.stdout:54 files removed 2024-05-08T00:26:15.662 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-05-08T00:26:15.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-05-08T00:26:15.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw 2024-05-08T00:26:15.903 INFO:teuthology.orchestra.run.smithi064.stdout:54 files removed 2024-05-08T00:26:15.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-05-08T00:26:16.509 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 150 kB/s | 83 kB 00:00 2024-05-08T00:26:16.615 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 211 kB/s | 83 kB 00:00 2024-05-08T00:26:17.299 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 18 kB/s | 11 kB 00:00 2024-05-08T00:26:17.933 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-05-08T00:26:18.060 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 7.7 kB/s | 11 kB 00:01 2024-05-08T00:26:18.419 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-05-08T00:26:19.269 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.1 MB 00:00 2024-05-08T00:26:19.725 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.1 MB 00:01 2024-05-08T00:26:22.471 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 19 MB 00:02 2024-05-08T00:26:22.601 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-05-08T00:26:27.696 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.2 MB 00:01 2024-05-08T00:26:28.027 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.2 MB 00:01 2024-05-08T00:26:29.122 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 16 kB 00:00 2024-05-08T00:26:29.703 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-05-08T00:26:31.094 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-08T00:26:33.321 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 21 MB 00:03 2024-05-08T00:26:36.282 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-05-08T00:26:38.639 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:26:38.657 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:26:38.657 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:26:38.657 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 10 M 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 3.4 M 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 3.1 M 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 5.4 M 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 21 M 2024-05-08T00:26:38.658 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 25 k 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 746 k 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 501 k 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 5.3 M 2024-05-08T00:26:38.659 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 45 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 135 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 161 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 321 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 301 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 99 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-08T00:26:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-08T00:26:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-08T00:26:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-05-08T00:26:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-08T00:26:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:26:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:26:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:26:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:Install 30 Packages 2024-05-08T00:26:38.661 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 2 Packages 2024-05-08T00:26:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:26:38.662 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 60 M 2024-05-08T00:26:38.662 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:26:38.796 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-08T00:26:39.492 INFO:teuthology.orchestra.run.smithi116.stdout:(1/32): ceph-base-19.0.0-2529.ge0c1691a.el9.x86 8.3 MB/s | 5.4 MB 00:00 2024-05-08T00:26:39.542 INFO:teuthology.orchestra.run.smithi116.stdout:(2/32): ceph-selinux-19.0.0-2529.ge0c1691a.el9. 503 kB/s | 25 kB 00:00 2024-05-08T00:26:39.642 INFO:teuthology.orchestra.run.smithi116.stdout:(3/32): libcephfs2-19.0.0-2529.ge0c1691a.el9.x8 7.3 MB/s | 746 kB 00:00 2024-05-08T00:26:39.718 INFO:teuthology.orchestra.run.smithi116.stdout:(4/32): libradosstriper1-19.0.0-2529.ge0c1691a. 6.5 MB/s | 501 kB 00:00 2024-05-08T00:26:40.251 INFO:teuthology.orchestra.run.smithi116.stdout:(5/32): librgw2-19.0.0-2529.ge0c1691a.el9.x86_6 10 MB/s | 5.3 MB 00:00 2024-05-08T00:26:40.360 INFO:teuthology.orchestra.run.smithi116.stdout:(6/32): ceph-radosgw-19.0.0-2529.ge0c1691a.el9. 6.8 MB/s | 10 MB 00:01 2024-05-08T00:26:40.385 INFO:teuthology.orchestra.run.smithi116.stdout:(7/32): python3-ceph-argparse-19.0.0-2529.ge0c1 338 kB/s | 45 kB 00:00 2024-05-08T00:26:40.410 INFO:teuthology.orchestra.run.smithi116.stdout:(8/32): python3-ceph-common-19.0.0-2529.ge0c169 2.6 MB/s | 135 kB 00:00 2024-05-08T00:26:40.435 INFO:teuthology.orchestra.run.smithi116.stdout:(9/32): python3-cephfs-19.0.0-2529.ge0c1691a.el 3.1 MB/s | 161 kB 00:00 2024-05-08T00:26:40.461 INFO:teuthology.orchestra.run.smithi116.stdout:(10/32): python3-rados-19.0.0-2529.ge0c1691a.el 6.2 MB/s | 321 kB 00:00 2024-05-08T00:26:40.486 INFO:teuthology.orchestra.run.smithi116.stdout:(11/32): python3-rbd-19.0.0-2529.ge0c1691a.el9. 5.8 MB/s | 301 kB 00:00 2024-05-08T00:26:40.511 INFO:teuthology.orchestra.run.smithi116.stdout:(12/32): python3-rgw-19.0.0-2529.ge0c1691a.el9. 1.9 MB/s | 99 kB 00:00 2024-05-08T00:26:40.637 INFO:teuthology.orchestra.run.smithi116.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 712 kB/s | 107 kB 00:00 2024-05-08T00:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-05-08T00:26:40.679 INFO:teuthology.orchestra.run.smithi116.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-05-08T00:26:40.737 INFO:teuthology.orchestra.run.smithi116.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-05-08T00:26:40.763 INFO:teuthology.orchestra.run.smithi116.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-05-08T00:26:40.788 INFO:teuthology.orchestra.run.smithi116.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-05-08T00:26:40.813 INFO:teuthology.orchestra.run.smithi116.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-08T00:26:40.839 INFO:teuthology.orchestra.run.smithi116.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 720 kB/s | 36 kB 00:00 2024-05-08T00:26:40.864 INFO:teuthology.orchestra.run.smithi116.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-05-08T00:26:41.257 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:26:41.265 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:26:41.265 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:26:41.265 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 10 M 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 3.4 M 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 3.1 M 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 5.4 M 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 21 M 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 25 k 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-08T00:26:41.266 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 746 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 501 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 5.3 M 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 45 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 135 k 2024-05-08T00:26:41.267 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 161 k 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 321 k 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 301 k 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 99 k 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout:Install 30 Packages 2024-05-08T00:26:41.268 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-05-08T00:26:41.269 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:26:41.269 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 60 M 2024-05-08T00:26:41.269 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:26:41.381 INFO:teuthology.orchestra.run.smithi116.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 570 kB/s | 309 kB 00:00 2024-05-08T00:26:41.481 INFO:teuthology.orchestra.run.smithi116.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-05-08T00:26:41.582 INFO:teuthology.orchestra.run.smithi116.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-05-08T00:26:41.690 INFO:teuthology.orchestra.run.smithi116.stdout:(25/32): libarrow-9.0.0-11.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-05-08T00:26:41.716 INFO:teuthology.orchestra.run.smithi116.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-05-08T00:26:41.813 INFO:teuthology.orchestra.run.smithi116.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.7 MB/s | 839 kB 00:00 2024-05-08T00:26:41.841 INFO:teuthology.orchestra.run.smithi116.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-05-08T00:26:42.058 INFO:teuthology.orchestra.run.smithi116.stdout:(29/32): ceph-common-19.0.0-2529.ge0c1691a.el9. 6.5 MB/s | 21 MB 00:03 2024-05-08T00:26:42.100 INFO:teuthology.orchestra.run.smithi116.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.5 MB/s | 1.6 MB 00:00 2024-05-08T00:26:42.617 INFO:teuthology.orchestra.run.smithi116.stdout:(31/32): librados2-19.0.0-2529.ge0c1691a.el9.x8 4.4 MB/s | 3.4 MB 00:00 2024-05-08T00:26:42.784 INFO:teuthology.orchestra.run.smithi116.stdout:(32/32): librbd1-19.0.0-2529.ge0c1691a.el9.x86_ 4.3 MB/s | 3.1 MB 00:00 2024-05-08T00:26:42.788 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:26:42.789 INFO:teuthology.orchestra.run.smithi116.stdout:Total 14 MB/s | 60 MB 00:04 2024-05-08T00:26:42.895 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:26:42.930 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:26:42.930 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:26:43.072 INFO:teuthology.orchestra.run.smithi064.stdout:(1/32): ceph-radosgw-19.0.0-2529.ge0c1691a.el9. 7.1 MB/s | 10 MB 00:01 2024-05-08T00:26:43.122 INFO:teuthology.orchestra.run.smithi064.stdout:(2/32): ceph-selinux-19.0.0-2529.ge0c1691a.el9. 504 kB/s | 25 kB 00:00 2024-05-08T00:26:43.231 INFO:teuthology.orchestra.run.smithi064.stdout:(3/32): ceph-base-19.0.0-2529.ge0c1691a.el9.x86 3.3 MB/s | 5.4 MB 00:01 2024-05-08T00:26:43.264 INFO:teuthology.orchestra.run.smithi064.stdout:(4/32): libcephfs2-19.0.0-2529.ge0c1691a.el9.x8 5.1 MB/s | 746 kB 00:00 2024-05-08T00:26:43.348 INFO:teuthology.orchestra.run.smithi064.stdout:(5/32): libradosstriper1-19.0.0-2529.ge0c1691a. 4.2 MB/s | 501 kB 00:00 2024-05-08T00:26:43.424 INFO:teuthology.orchestra.run.smithi064.stdout:(6/32): python3-ceph-argparse-19.0.0-2529.ge0c1 601 kB/s | 45 kB 00:00 2024-05-08T00:26:43.499 INFO:teuthology.orchestra.run.smithi064.stdout:(7/32): python3-ceph-common-19.0.0-2529.ge0c169 1.8 MB/s | 135 kB 00:00 2024-05-08T00:26:43.582 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:26:43.582 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:26:43.583 INFO:teuthology.orchestra.run.smithi064.stdout:(8/32): python3-cephfs-19.0.0-2529.ge0c1691a.el 1.9 MB/s | 161 kB 00:00 2024-05-08T00:26:43.700 INFO:teuthology.orchestra.run.smithi064.stdout:(9/32): python3-rados-19.0.0-2529.ge0c1691a.el9 2.7 MB/s | 321 kB 00:00 2024-05-08T00:26:43.825 INFO:teuthology.orchestra.run.smithi064.stdout:(10/32): python3-rbd-19.0.0-2529.ge0c1691a.el9. 2.4 MB/s | 301 kB 00:00 2024-05-08T00:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout:(11/32): ceph-common-19.0.0-2529.ge0c1691a.el9. 8.7 MB/s | 21 MB 00:02 2024-05-08T00:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout:(12/32): python3-rgw-19.0.0-2529.ge0c1691a.el9. 411 kB/s | 99 kB 00:00 2024-05-08T00:26:44.134 INFO:teuthology.orchestra.run.smithi064.stdout:(13/32): librgw2-19.0.0-2529.ge0c1691a.el9.x86_ 6.1 MB/s | 5.3 MB 00:00 2024-05-08T00:26:44.759 INFO:teuthology.orchestra.run.smithi064.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-05-08T00:26:44.784 INFO:teuthology.orchestra.run.smithi064.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 144 kB/s | 107 kB 00:00 2024-05-08T00:26:44.851 INFO:teuthology.orchestra.run.smithi064.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 204 kB/s | 160 kB 00:00 2024-05-08T00:26:44.965 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:26:45.035 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-08T00:26:45.063 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-08T00:26:45.097 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-08T00:26:45.120 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-08T00:26:45.142 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-08T00:26:45.345 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-08T00:26:45.406 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-08T00:26:45.543 INFO:teuthology.orchestra.run.smithi064.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 61 kB/s | 42 kB 00:00 2024-05-08T00:26:45.652 INFO:teuthology.orchestra.run.smithi064.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 337 kB/s | 292 kB 00:00 2024-05-08T00:26:45.719 INFO:teuthology.orchestra.run.smithi064.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 691 kB/s | 662 kB 00:00 2024-05-08T00:26:45.737 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-08T00:26:45.832 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 7/34 2024-05-08T00:26:45.923 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 7/34 2024-05-08T00:26:46.011 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-2529.ge0c1691a.el9.x86_64 8/34 2024-05-08T00:26:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 9/34 2024-05-08T00:26:46.097 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 9/34 2024-05-08T00:26:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-08T00:26:46.168 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-2529.ge0c1691a.el9. 11/34 2024-05-08T00:26:46.236 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-2529.ge0c1691a.el9.x86_64 12/34 2024-05-08T00:26:46.244 INFO:teuthology.orchestra.run.smithi064.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 61 kB/s | 36 kB 00:00 2024-05-08T00:26:46.261 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-2529.ge0c1691a.el9.x86_6 13/34 2024-05-08T00:26:46.308 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2529.ge0c1691a.el9.x86_6 13/34 2024-05-08T00:26:46.334 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-08T00:26:46.336 INFO:teuthology.orchestra.run.smithi064.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 77 kB/s | 47 kB 00:00 2024-05-08T00:26:46.369 INFO:teuthology.orchestra.run.smithi064.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 267 kB/s | 221 kB 00:00 2024-05-08T00:26:46.449 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-08T00:26:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-08T00:26:46.503 INFO:teuthology.orchestra.run.smithi064.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-05-08T00:26:46.526 INFO:teuthology.orchestra.run.smithi064.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 166 kB/s | 26 kB 00:00 2024-05-08T00:26:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-08T00:26:46.553 INFO:teuthology.orchestra.run.smithi064.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 964 kB/s | 49 kB 00:00 2024-05-08T00:26:46.559 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-08T00:26:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-08T00:26:46.620 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-08T00:26:46.654 INFO:teuthology.orchestra.run.smithi064.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 530 kB/s | 67 kB 00:00 2024-05-08T00:26:46.687 INFO:teuthology.orchestra.run.smithi064.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-05-08T00:26:46.796 INFO:teuthology.orchestra.run.smithi064.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-05-08T00:26:46.838 INFO:teuthology.orchestra.run.smithi064.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-05-08T00:26:46.905 INFO:teuthology.orchestra.run.smithi064.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-05-08T00:26:46.930 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-08T00:26:46.975 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 22/34 2024-05-08T00:26:47.070 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 22/34 2024-05-08T00:26:47.097 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 23/34 2024-05-08T00:26:47.133 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-08T00:26:47.138 INFO:teuthology.orchestra.run.smithi064.stdout:(31/32): librbd1-19.0.0-2529.ge0c1691a.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-05-08T00:26:47.430 INFO:teuthology.orchestra.run.smithi064.stdout:(32/32): librados2-19.0.0-2529.ge0c1691a.el9.x8 5.3 MB/s | 3.4 MB 00:00 2024-05-08T00:26:47.433 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:26:47.433 INFO:teuthology.orchestra.run.smithi064.stdout:Total 9.7 MB/s | 60 MB 00:06 2024-05-08T00:26:47.544 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:26:47.586 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:26:47.586 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:26:47.609 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-08T00:26:47.644 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-2529.ge0c1691a.el9.x86_64 26/34 2024-05-08T00:26:47.716 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-2529.ge0c1691a.el9.x86_64 26/34 2024-05-08T00:26:47.794 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 27/34 2024-05-08T00:26:47.847 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-2529.ge0c1691a.el9.x8 28/34 2024-05-08T00:26:48.236 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:26:48.236 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:26:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:26:49.779 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-08T00:26:49.811 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-08T00:26:49.844 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-08T00:26:49.867 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-08T00:26:49.888 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-08T00:26:50.077 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 29/34 2024-05-08T00:26:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-08T00:26:50.110 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 29/34 2024-05-08T00:26:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-08T00:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-08T00:26:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 7/34 2024-05-08T00:26:50.679 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 29/34 2024-05-08T00:26:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 7/34 2024-05-08T00:26:50.714 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 30/34 2024-05-08T00:26:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-2529.ge0c1691a.el9.x86_64 8/34 2024-05-08T00:26:50.802 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 30/34 2024-05-08T00:26:50.802 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-08T00:26:50.802 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-08T00:26:50.803 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:26:50.805 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 9/34 2024-05-08T00:26:50.840 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-2529.ge0c1691a.el9.x86_64 31/34 2024-05-08T00:26:50.860 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 9/34 2024-05-08T00:26:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-08T00:26:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-2529.ge0c1691a.el9. 11/34 2024-05-08T00:26:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-2529.ge0c1691a.el9.x86_64 12/34 2024-05-08T00:26:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-2529.ge0c1691a.el9.x86_6 13/34 2024-05-08T00:26:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2529.ge0c1691a.el9.x86_6 13/34 2024-05-08T00:26:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-08T00:26:51.262 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-08T00:26:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-08T00:26:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-08T00:26:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-08T00:26:51.401 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-08T00:26:51.434 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-08T00:26:51.760 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-08T00:26:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 22/34 2024-05-08T00:26:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 22/34 2024-05-08T00:26:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 23/34 2024-05-08T00:26:51.955 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-08T00:26:52.462 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-08T00:26:52.491 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-2529.ge0c1691a.el9.x86_64 26/34 2024-05-08T00:26:52.563 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-2529.ge0c1691a.el9.x86_64 26/34 2024-05-08T00:26:52.643 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 27/34 2024-05-08T00:26:52.669 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-2529.ge0c1691a.el9.x8 28/34 2024-05-08T00:26:55.050 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 29/34 2024-05-08T00:26:55.082 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 29/34 2024-05-08T00:26:55.678 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 29/34 2024-05-08T00:26:55.727 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 30/34 2024-05-08T00:26:56.016 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 30/34 2024-05-08T00:26:56.016 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-08T00:26:56.016 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-08T00:26:56.016 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:26:56.054 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-2529.ge0c1691a.el9.x86_64 31/34 2024-05-08T00:27:00.514 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2529.ge0c1691a.el9.x86_64 31/34 2024-05-08T00:27:00.514 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-05-08T00:27:00.514 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-05-08T00:27:00.514 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-05-08T00:27:00.514 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-05-08T00:27:00.514 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-05-08T00:27:00.514 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-05-08T00:27:00.514 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-05-08T00:27:00.514 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 32/34 2024-05-08T00:27:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 32/34 2024-05-08T00:27:01.449 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:01.449 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-08T00:27:01.449 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-08T00:27:01.449 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-08T00:27:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:01.451 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-08T00:27:01.535 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-08T00:27:01.535 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-08T00:27:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 3/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-2529.ge0c1691a.el9.x86_64 4/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 5/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-2529.ge0c1691a.el9.x86_6 6/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-2529.ge0c1691a.el9.x86_64 7/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2529.ge0c1691a.el9. 8/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-2529.ge0c1691a.el9.x8 9/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-2529.ge0c1691a.el9.x86_64 10/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-2529.ge0c1691a.el9.x86_64 11/34 2024-05-08T00:27:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 12/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 13/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-08T00:27:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-08T00:27:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-08T00:27:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-08T00:27:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-08T00:27:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 31/34 2024-05-08T00:27:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-08T00:27:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 33/34 2024-05-08T00:27:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-08T00:27:03.398 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.399 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.400 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-08T00:27:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-08T00:27:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-08T00:27:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-05-08T00:27:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-08T00:27:03.401 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:03.401 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:27:03.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-test 2024-05-08T00:27:04.306 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:28 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:27:04.426 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:27:04.427 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:04.427 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:27:04.427 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:04.427 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:27:04.427 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 48 M 2024-05-08T00:27:04.427 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:27:04.427 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 164 k 2024-05-08T00:27:04.427 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-08T00:27:04.428 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-08T00:27:04.428 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:04.428 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:27:04.428 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:04.428 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-05-08T00:27:04.428 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:04.428 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 49 M 2024-05-08T00:27:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 207 M 2024-05-08T00:27:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:27:04.680 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 305 kB 00:00 2024-05-08T00:27:04.739 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-08T00:27:04.864 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): libcephsqlite-19.0.0-2529.ge0c1691a.el9. 411 kB/s | 164 kB 00:00 2024-05-08T00:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2529.ge0c1691a.el9.x86_64 31/34 2024-05-08T00:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-05-08T00:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-05-08T00:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-05-08T00:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-05-08T00:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-05-08T00:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-05-08T00:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-05-08T00:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:06.780 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 32/34 2024-05-08T00:27:06.807 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 32/34 2024-05-08T00:27:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-08T00:27:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-08T00:27:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-08T00:27:06.807 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:06.809 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-08T00:27:06.883 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-08T00:27:06.883 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-08T00:27:07.015 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): ceph-test-19.0.0-2529.ge0c1691a.el9.x86_ 19 MB/s | 48 MB 00:02 2024-05-08T00:27:07.017 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:07.017 INFO:teuthology.orchestra.run.smithi116.stdout:Total 19 MB/s | 49 MB 00:02 2024-05-08T00:27:07.045 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:27:07.065 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:27:07.066 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:27:07.429 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:27:07.429 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:27:07.950 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:27:08.032 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-08T00:27:08.072 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-08T00:27:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-08T00:27:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/34 2024-05-08T00:27:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/34 2024-05-08T00:27:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 3/34 2024-05-08T00:27:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-2529.ge0c1691a.el9.x86_64 4/34 2024-05-08T00:27:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 5/34 2024-05-08T00:27:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-2529.ge0c1691a.el9.x86_6 6/34 2024-05-08T00:27:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-2529.ge0c1691a.el9.x86_64 7/34 2024-05-08T00:27:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2529.ge0c1691a.el9. 8/34 2024-05-08T00:27:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-2529.ge0c1691a.el9.x8 9/34 2024-05-08T00:27:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-2529.ge0c1691a.el9.x86_64 10/34 2024-05-08T00:27:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-2529.ge0c1691a.el9.x86_64 11/34 2024-05-08T00:27:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 12/34 2024-05-08T00:27:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 13/34 2024-05-08T00:27:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-08T00:27:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-08T00:27:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-08T00:27:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-08T00:27:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-08T00:27:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-08T00:27:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-08T00:27:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-08T00:27:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-08T00:27:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-08T00:27:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-08T00:27:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-08T00:27:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-08T00:27:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-08T00:27:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-08T00:27:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-08T00:27:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-08T00:27:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 31/34 2024-05-08T00:27:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-08T00:27:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 33/34 2024-05-08T00:27:08.100 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-2529.ge0c1691a.el9.x86_64 3/4 2024-05-08T00:27:08.811 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.812 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.813 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:08.814 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:27:09.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-05-08T00:27:09.777 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:31 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:27:09.903 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:27:09.903 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 48 M 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 164 k 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-05-08T00:27:09.904 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:09.905 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 49 M 2024-05-08T00:27:09.905 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 207 M 2024-05-08T00:27:09.905 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:27:10.220 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 305 kB 00:00 2024-05-08T00:27:10.262 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-05-08T00:27:10.387 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): libcephsqlite-19.0.0-2529.ge0c1691a.el9. 450 kB/s | 164 kB 00:00 2024-05-08T00:27:12.538 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): ceph-test-19.0.0-2529.ge0c1691a.el9.x86_ 19 MB/s | 48 MB 00:02 2024-05-08T00:27:12.540 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:12.540 INFO:teuthology.orchestra.run.smithi064.stdout:Total 19 MB/s | 49 MB 00:02 2024-05-08T00:27:12.569 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:27:12.592 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:27:12.592 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:27:12.741 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2529.ge0c1691a.el9.x86_64 3/4 2024-05-08T00:27:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-2529.ge0c1691a.el9.x86_64 4/4 2024-05-08T00:27:12.978 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:27:12.978 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:27:13.526 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:27:13.615 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-08T00:27:13.654 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-08T00:27:13.681 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-2529.ge0c1691a.el9.x86_64 3/4 2024-05-08T00:27:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-test-2:19.0.0-2529.ge0c1691a.el9.x86_64 4/4 2024-05-08T00:27:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/4 2024-05-08T00:27:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/4 2024-05-08T00:27:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-08T00:27:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-08T00:27:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:15.318 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:27:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-08T00:27:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-08T00:27:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:15.319 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:27:15.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph 2024-05-08T00:27:16.142 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:40 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:27:16.267 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:27:16.269 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 6.6 k 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 2.4 M 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 1.5 M 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 249 k 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 4.7 M 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 17 M 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-08T00:27:16.270 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-08T00:27:16.271 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:16.272 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:27:16.273 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:16.273 INFO:teuthology.orchestra.run.smithi116.stdout:Install 36 Packages 2024-05-08T00:27:16.273 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:16.273 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 30 M 2024-05-08T00:27:16.273 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 107 M 2024-05-08T00:27:16.273 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:27:17.075 INFO:teuthology.orchestra.run.smithi116.stdout:(1/36): ceph-19.0.0-2529.ge0c1691a.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-05-08T00:27:18.108 INFO:teuthology.orchestra.run.smithi116.stdout:(2/36): ceph-mon-19.0.0-2529.ge0c1691a.el9.x86_ 4.5 MB/s | 4.7 MB 00:01 2024-05-08T00:27:18.142 INFO:teuthology.orchestra.run.smithi116.stdout:(3/36): ceph-mgr-19.0.0-2529.ge0c1691a.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-05-08T00:27:18.334 INFO:teuthology.orchestra.run.smithi116.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2529.ge0c1 1.3 MB/s | 249 kB 00:00 2024-05-08T00:27:18.409 INFO:teuthology.orchestra.run.smithi116.stdout:(5/36): ceph-mds-19.0.0-2529.ge0c1691a.el9.x86_ 1.6 MB/s | 2.4 MB 00:01 2024-05-08T00:27:18.501 INFO:teuthology.orchestra.run.smithi116.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-05-08T00:27:18.551 INFO:teuthology.orchestra.run.smithi116.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-08T00:27:18.560 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2529.ge0c1691a.el9.x86_64 3/4 2024-05-08T00:27:18.626 INFO:teuthology.orchestra.run.smithi116.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.8 MB/s | 135 kB 00:00 2024-05-08T00:27:18.649 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-2529.ge0c1691a.el9.x86_64 4/4 2024-05-08T00:27:18.660 INFO:teuthology.orchestra.run.smithi116.stdout:(9/36): python3-cryptography-36.0.1-4.el9.x86_6 4.9 MB/s | 1.2 MB 00:00 2024-05-08T00:27:18.735 INFO:teuthology.orchestra.run.smithi116.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-05-08T00:27:18.760 INFO:teuthology.orchestra.run.smithi116.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-05-08T00:27:18.969 INFO:teuthology.orchestra.run.smithi116.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 827 kB/s | 172 kB 00:00 2024-05-08T00:27:18.994 INFO:teuthology.orchestra.run.smithi116.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 728 kB/s | 188 kB 00:00 2024-05-08T00:27:19.153 INFO:teuthology.orchestra.run.smithi116.stdout:(14/36): ceph-osd-19.0.0-2529.ge0c1691a.el9.x86 16 MB/s | 17 MB 00:01 2024-05-08T00:27:19.170 INFO:teuthology.orchestra.run.smithi116.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 173 kB/s | 35 kB 00:00 2024-05-08T00:27:19.187 INFO:teuthology.orchestra.run.smithi116.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 265 kB/s | 51 kB 00:00 2024-05-08T00:27:19.237 INFO:teuthology.orchestra.run.smithi116.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 332 kB/s | 22 kB 00:00 2024-05-08T00:27:19.321 INFO:teuthology.orchestra.run.smithi116.stdout:(18/36): python3-toml-0.10.2-6.el9.noarch.rpm 249 kB/s | 42 kB 00:00 2024-05-08T00:27:19.379 INFO:teuthology.orchestra.run.smithi116.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 306 kB/s | 43 kB 00:00 2024-05-08T00:27:19.422 INFO:teuthology.orchestra.run.smithi116.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 645 kB/s | 151 kB 00:00 2024-05-08T00:27:19.472 INFO:teuthology.orchestra.run.smithi116.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-05-08T00:27:19.530 INFO:teuthology.orchestra.run.smithi116.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-05-08T00:27:19.556 INFO:teuthology.orchestra.run.smithi116.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 733 kB/s | 172 kB 00:00 2024-05-08T00:27:19.581 INFO:teuthology.orchestra.run.smithi116.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-05-08T00:27:19.606 INFO:teuthology.orchestra.run.smithi116.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-05-08T00:27:19.631 INFO:teuthology.orchestra.run.smithi116.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-05-08T00:27:19.656 INFO:teuthology.orchestra.run.smithi116.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-05-08T00:27:19.682 INFO:teuthology.orchestra.run.smithi116.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-05-08T00:27:19.707 INFO:teuthology.orchestra.run.smithi116.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-05-08T00:27:19.732 INFO:teuthology.orchestra.run.smithi116.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-05-08T00:27:19.757 INFO:teuthology.orchestra.run.smithi116.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-05-08T00:27:19.783 INFO:teuthology.orchestra.run.smithi116.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-05-08T00:27:19.808 INFO:teuthology.orchestra.run.smithi116.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-05-08T00:27:19.850 INFO:teuthology.orchestra.run.smithi116.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-05-08T00:27:19.875 INFO:teuthology.orchestra.run.smithi116.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-05-08T00:27:19.909 INFO:teuthology.orchestra.run.smithi116.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-05-08T00:27:19.910 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:19.910 INFO:teuthology.orchestra.run.smithi116.stdout:Total 8.3 MB/s | 30 MB 00:03 2024-05-08T00:27:20.055 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:27:20.093 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:27:20.093 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:27:20.534 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:27:20.534 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:27:21.005 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.0.0-2529.ge0c1691a.el9.x86_64 4/4 2024-05-08T00:27:21.005 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/4 2024-05-08T00:27:21.005 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/4 2024-05-08T00:27:21.005 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-08T00:27:21.288 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-08T00:27:21.288 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:21.288 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:27:21.288 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:21.288 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:21.289 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-08T00:27:21.289 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-08T00:27:21.289 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:21.289 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:27:21.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-05-08T00:27:21.775 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:27:21.828 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-08T00:27:21.861 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-08T00:27:21.888 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-08T00:27:21.961 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-08T00:27:22.055 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-08T00:27:22.088 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-08T00:27:22.176 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:44 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:27:22.181 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-08T00:27:22.258 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-08T00:27:22.298 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-08T00:27:22.308 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:27:22.311 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 6.6 k 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 2.4 M 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 1.5 M 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 249 k 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 4.7 M 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 17 M 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-08T00:27:22.312 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-08T00:27:22.313 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-08T00:27:22.314 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:22.315 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:27:22.315 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:22.315 INFO:teuthology.orchestra.run.smithi064.stdout:Install 36 Packages 2024-05-08T00:27:22.315 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:22.315 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 30 M 2024-05-08T00:27:22.315 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 107 M 2024-05-08T00:27:22.315 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:27:22.333 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-08T00:27:22.363 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-08T00:27:22.397 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-08T00:27:22.431 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-08T00:27:22.463 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-08T00:27:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-08T00:27:22.533 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-08T00:27:22.571 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-08T00:27:22.645 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-08T00:27:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-08T00:27:22.863 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-08T00:27:22.916 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-08T00:27:22.954 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-08T00:27:23.011 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-08T00:27:23.031 INFO:teuthology.orchestra.run.smithi064.stdout:(1/36): ceph-19.0.0-2529.ge0c1691a.el9.x86_64.r 34 kB/s | 6.6 kB 00:00 2024-05-08T00:27:23.075 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-08T00:27:23.231 INFO:teuthology.orchestra.run.smithi064.stdout:(2/36): ceph-mds-19.0.0-2529.ge0c1691a.el9.x86_ 6.0 MB/s | 2.4 MB 00:00 2024-05-08T00:27:23.267 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-08T00:27:23.273 INFO:teuthology.orchestra.run.smithi064.stdout:(3/36): ceph-mgr-19.0.0-2529.ge0c1691a.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-05-08T00:27:23.316 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-08T00:27:23.349 INFO:teuthology.orchestra.run.smithi064.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2529.ge0c1 3.2 MB/s | 249 kB 00:00 2024-05-08T00:27:23.403 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-08T00:27:23.492 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-08T00:27:23.532 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-08T00:27:23.648 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-08T00:27:23.807 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2529.ge0c1691a.el9. 31/36 2024-05-08T00:27:23.842 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 32/36 2024-05-08T00:27:23.867 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 32/36 2024-05-08T00:27:23.867 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:23.868 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-08T00:27:23.868 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-08T00:27:23.868 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-08T00:27:23.868 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:24.091 INFO:teuthology.orchestra.run.smithi064.stdout:(5/36): ceph-osd-19.0.0-2529.ge0c1691a.el9.x86_ 20 MB/s | 17 MB 00:00 2024-05-08T00:27:24.216 INFO:teuthology.orchestra.run.smithi064.stdout:(6/36): ceph-mon-19.0.0-2529.ge0c1691a.el9.x86_ 4.0 MB/s | 4.7 MB 00:01 2024-05-08T00:27:24.241 INFO:teuthology.orchestra.run.smithi064.stdout:(7/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 284 kB/s | 253 kB 00:00 2024-05-08T00:27:24.950 INFO:teuthology.orchestra.run.smithi064.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-05-08T00:27:24.975 INFO:teuthology.orchestra.run.smithi064.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 184 kB/s | 135 kB 00:00 2024-05-08T00:27:25.158 INFO:teuthology.orchestra.run.smithi064.stdout:(10/36): python3-cryptography-36.0.1-4.el9.x86_ 1.1 MB/s | 1.2 MB 00:01 2024-05-08T00:27:25.337 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-2529.ge0c1691a.el9.x86_64 33/36 2024-05-08T00:27:25.367 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-2529.ge0c1691a.el9.x86_64 33/36 2024-05-08T00:27:25.367 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:25.367 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-08T00:27:25.367 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-08T00:27:25.367 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-08T00:27:25.367 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:25.700 INFO:teuthology.orchestra.run.smithi064.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 166 kB/s | 125 kB 00:00 2024-05-08T00:27:25.759 INFO:teuthology.orchestra.run.smithi064.stdout:(12/36): python3-urllib3-1.26.5-5.el9.noarch.rp 274 kB/s | 215 kB 00:00 2024-05-08T00:27:25.822 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-2529.ge0c1691a.el9.x86_64 34/36 2024-05-08T00:27:25.846 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-2529.ge0c1691a.el9.x86_64 34/36 2024-05-08T00:27:25.846 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:25.847 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-08T00:27:25.847 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-08T00:27:25.847 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-08T00:27:25.847 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:25.967 INFO:teuthology.orchestra.run.smithi064.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 233 kB/s | 188 kB 00:00 2024-05-08T00:27:26.081 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-2529.ge0c1691a.el9.x86_64 35/36 2024-05-08T00:27:26.106 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-2529.ge0c1691a.el9.x86_64 35/36 2024-05-08T00:27:26.106 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:26.106 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-08T00:27:26.106 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-08T00:27:26.106 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-08T00:27:26.106 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:26.170 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-2529.ge0c1691a.el9.x86_64 36/36 2024-05-08T00:27:26.360 INFO:teuthology.orchestra.run.smithi064.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 58 kB/s | 35 kB 00:00 2024-05-08T00:27:26.468 INFO:teuthology.orchestra.run.smithi064.stdout:(15/36): python3-mako-1.1.4-6.el9.noarch.rpm 224 kB/s | 172 kB 00:00 2024-05-08T00:27:26.518 INFO:teuthology.orchestra.run.smithi064.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-05-08T00:27:26.602 INFO:teuthology.orchestra.run.smithi064.stdout:(17/36): python3-pytz-2021.1-5.el9.noarch.rpm 80 kB/s | 51 kB 00:00 2024-05-08T00:27:26.810 INFO:teuthology.orchestra.run.smithi064.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 518 kB/s | 151 kB 00:00 2024-05-08T00:27:26.877 INFO:teuthology.orchestra.run.smithi064.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 176 kB/s | 43 kB 00:00 2024-05-08T00:27:26.902 INFO:teuthology.orchestra.run.smithi064.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-05-08T00:27:26.961 INFO:teuthology.orchestra.run.smithi064.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-05-08T00:27:27.020 INFO:teuthology.orchestra.run.smithi064.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-05-08T00:27:27.045 INFO:teuthology.orchestra.run.smithi064.stdout:(23/36): python3-toml-0.10.2-6.el9.noarch.rpm 61 kB/s | 42 kB 00:00 2024-05-08T00:27:27.070 INFO:teuthology.orchestra.run.smithi064.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-05-08T00:27:27.095 INFO:teuthology.orchestra.run.smithi064.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 311 kB/s | 23 kB 00:00 2024-05-08T00:27:27.146 INFO:teuthology.orchestra.run.smithi064.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-05-08T00:27:27.179 INFO:teuthology.orchestra.run.smithi064.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 847 kB/s | 46 kB 00:00 2024-05-08T00:27:27.205 INFO:teuthology.orchestra.run.smithi064.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-05-08T00:27:27.263 INFO:teuthology.orchestra.run.smithi064.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-05-08T00:27:27.289 INFO:teuthology.orchestra.run.smithi064.stdout:(30/36): python3-jaraco-functools-3.5.0-2.el9.n 80 kB/s | 19 kB 00:00 2024-05-08T00:27:27.314 INFO:teuthology.orchestra.run.smithi064.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-05-08T00:27:27.339 INFO:teuthology.orchestra.run.smithi064.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-05-08T00:27:27.381 INFO:teuthology.orchestra.run.smithi064.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2024-05-08T00:27:27.440 INFO:teuthology.orchestra.run.smithi064.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-05-08T00:27:27.465 INFO:teuthology.orchestra.run.smithi064.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-05-08T00:27:27.500 INFO:teuthology.orchestra.run.smithi064.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 168 kB/s | 20 kB 00:00 2024-05-08T00:27:27.507 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:27.508 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.8 MB/s | 30 MB 00:05 2024-05-08T00:27:27.657 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:27:27.701 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:27:27.702 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:27:27.870 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-2:19.0.0-2529.ge0c1691a.el9.x86_64 36/36 2024-05-08T00:27:27.870 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/36 2024-05-08T00:27:27.870 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/36 2024-05-08T00:27:27.870 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 3/36 2024-05-08T00:27:27.870 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-2529.ge0c1691a.el9.x86_64 4/36 2024-05-08T00:27:27.870 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-2529.ge0c1691a.el9.x86_64 5/36 2024-05-08T00:27:27.870 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2529.ge0c1691a.el9. 6/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-08T00:27:27.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-08T00:27:27.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-08T00:27:27.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-08T00:27:27.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-08T00:27:27.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-08T00:27:27.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-08T00:27:28.188 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:27:28.188 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-08T00:27:28.766 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-08T00:27:28.767 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-08T00:27:28.768 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:28.769 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:27:29.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-base 2024-05-08T00:27:29.460 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:27:29.518 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-08T00:27:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-08T00:27:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-08T00:27:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-08T00:27:29.684 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:53 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:27:29.695 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-08T00:27:29.727 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-08T00:27:29.765 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:27:29.808 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:27:29.809 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:27:29.809 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:27:29.823 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-08T00:27:29.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install cephadm 2024-05-08T00:27:29.908 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-08T00:27:29.955 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-08T00:27:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-08T00:27:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-08T00:27:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-08T00:27:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-08T00:27:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-08T00:27:30.181 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-08T00:27:30.214 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-08T00:27:30.253 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-08T00:27:30.329 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-08T00:27:30.443 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:54 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:27:30.481 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-08T00:27:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-08T00:27:30.568 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:27:30.569 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:30.569 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:27:30.569 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:30.569 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:27:30.569 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 763 k 2024-05-08T00:27:30.569 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:30.569 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:27:30.569 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:30.570 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-05-08T00:27:30.570 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:30.570 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 763 k 2024-05-08T00:27:30.570 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 769 k 2024-05-08T00:27:30.570 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:27:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-08T00:27:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-08T00:27:30.719 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-08T00:27:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-08T00:27:30.998 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-08T00:27:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-08T00:27:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-08T00:27:31.218 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-08T00:27:31.267 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-08T00:27:31.355 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm-19.0.0-2529.ge0c1691a.el9.noarch.rpm 972 kB/s | 763 kB 00:00 2024-05-08T00:27:31.356 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:31.356 INFO:teuthology.orchestra.run.smithi116.stdout:Total 970 kB/s | 763 kB 00:00 2024-05-08T00:27:31.356 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:27:31.362 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:27:31.362 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:27:31.368 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:27:31.368 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:27:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-08T00:27:31.440 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:27:31.560 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2529.ge0c1691a.el9. 31/36 2024-05-08T00:27:31.590 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 32/36 2024-05-08T00:27:31.618 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 32/36 2024-05-08T00:27:31.618 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:31.618 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-08T00:27:31.618 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-08T00:27:31.618 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-08T00:27:31.619 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:31.920 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/1 2024-05-08T00:27:32.027 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/1 2024-05-08T00:27:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/1 2024-05-08T00:27:32.625 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/1 2024-05-08T00:27:32.626 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:32.626 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:27:32.626 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:32.626 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:32.626 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:27:32.861 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-immutable-object-cache 2024-05-08T00:27:33.141 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-2529.ge0c1691a.el9.x86_64 33/36 2024-05-08T00:27:33.170 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-2529.ge0c1691a.el9.x86_64 33/36 2024-05-08T00:27:33.171 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:33.171 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-08T00:27:33.171 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-08T00:27:33.171 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-08T00:27:33.171 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:33.441 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:57 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:27:33.567 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:27:33.568 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:33.568 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-05-08T00:27:33.568 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 145 k 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 145 k 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 435 k 2024-05-08T00:27:33.569 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:27:33.642 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-2529.ge0c1691a.el9.x86_64 34/36 2024-05-08T00:27:33.667 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-2529.ge0c1691a.el9.x86_64 34/36 2024-05-08T00:27:33.667 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:33.668 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-08T00:27:33.668 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-08T00:27:33.668 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-08T00:27:33.668 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:33.901 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-2529.ge0c1691a.el9.x86_64 35/36 2024-05-08T00:27:33.926 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-2529.ge0c1691a.el9.x86_64 35/36 2024-05-08T00:27:33.926 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:33.926 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-08T00:27:33.926 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-08T00:27:33.927 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-08T00:27:33.927 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-2529.ge0c1691a.el9.x86_64 36/36 2024-05-08T00:27:34.094 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-immutable-object-cache-19.0.0-2529.ge0c169 276 kB/s | 145 kB 00:00 2024-05-08T00:27:34.096 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:34.096 INFO:teuthology.orchestra.run.smithi116.stdout:Total 275 kB/s | 145 kB 00:00 2024-05-08T00:27:34.096 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:27:34.104 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:27:34.104 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:27:34.162 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:27:34.162 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:27:34.306 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:27:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2529.ge0c1691a. 1/1 2024-05-08T00:27:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2529.ge0c1691a. 1/1 2024-05-08T00:27:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-08T00:27:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2529.ge0c1691a. 1/1 2024-05-08T00:27:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:35.326 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:27:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:35.326 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:27:35.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr 2024-05-08T00:27:35.752 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.0.0-2529.ge0c1691a.el9.x86_64 36/36 2024-05-08T00:27:35.752 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/36 2024-05-08T00:27:35.752 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/36 2024-05-08T00:27:35.752 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 3/36 2024-05-08T00:27:35.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-2529.ge0c1691a.el9.x86_64 4/36 2024-05-08T00:27:35.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-2529.ge0c1691a.el9.x86_64 5/36 2024-05-08T00:27:35.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2529.ge0c1691a.el9. 6/36 2024-05-08T00:27:35.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-08T00:27:35.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-08T00:27:35.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-08T00:27:35.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-08T00:27:35.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-08T00:27:35.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-08T00:27:35.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-08T00:27:35.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-08T00:27:35.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-08T00:27:35.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-08T00:27:35.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-08T00:27:35.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-08T00:27:35.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-08T00:27:35.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-08T00:27:35.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-08T00:27:35.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-08T00:27:36.128 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:00 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:27:36.212 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:27:36.254 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:27:36.255 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:27:36.255 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:27:36.312 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-dashboard 2024-05-08T00:27:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-08T00:27:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:36.662 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:27:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-08T00:27:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-08T00:27:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-08T00:27:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-08T00:27:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-08T00:27:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-08T00:27:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-08T00:27:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-08T00:27:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-08T00:27:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-08T00:27:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:36.665 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:27:36.880 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:00 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:27:37.003 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:27:37.004 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:27:37.004 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 3.5 M 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 32 k 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 16 k 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-08T00:27:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-08T00:27:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-08T00:27:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-08T00:27:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-08T00:27:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-08T00:27:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:37.006 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:27:37.006 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:37.006 INFO:teuthology.orchestra.run.smithi116.stdout:Install 12 Packages 2024-05-08T00:27:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:37.007 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 8.6 M 2024-05-08T00:27:37.007 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 103 M 2024-05-08T00:27:37.007 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:27:37.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-05-08T00:27:37.639 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:59 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:27:37.657 INFO:teuthology.orchestra.run.smithi116.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2529.ge0 164 kB/s | 32 kB 00:00 2024-05-08T00:27:37.723 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:27:37.757 INFO:teuthology.orchestra.run.smithi116.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2529.ge0c 56 kB/s | 16 kB 00:00 2024-05-08T00:27:37.768 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:27:37.769 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:27:37.769 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:27:37.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-05-08T00:27:38.249 INFO:teuthology.orchestra.run.smithi116.stdout:(3/12): ceph-mgr-dashboard-19.0.0-2529.ge0c1691 4.5 MB/s | 3.5 MB 00:00 2024-05-08T00:27:38.424 INFO:teuthology.orchestra.run.smithi116.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 404 kB/s | 269 kB 00:00 2024-05-08T00:27:38.437 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:00 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:27:38.458 INFO:teuthology.orchestra.run.smithi116.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.3 MB/s | 1.0 MB 00:00 2024-05-08T00:27:38.565 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:27:38.566 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:27:38.566 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:38.566 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:27:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 763 k 2024-05-08T00:27:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:38.566 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:27:38.566 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:38.567 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-05-08T00:27:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:38.567 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 763 k 2024-05-08T00:27:38.567 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 769 k 2024-05-08T00:27:38.567 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:27:38.708 INFO:teuthology.orchestra.run.smithi116.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.8 MB/s | 863 kB 00:00 2024-05-08T00:27:38.734 INFO:teuthology.orchestra.run.smithi116.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-05-08T00:27:38.992 INFO:teuthology.orchestra.run.smithi116.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 966 kB/s | 548 kB 00:00 2024-05-08T00:27:39.142 INFO:teuthology.orchestra.run.smithi116.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 354 kB/s | 144 kB 00:00 2024-05-08T00:27:39.243 INFO:teuthology.orchestra.run.smithi116.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 123 kB/s | 31 kB 00:00 2024-05-08T00:27:39.392 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.0.0-2529.ge0c1691a.el9.noarch.rpm 925 kB/s | 763 kB 00:00 2024-05-08T00:27:39.392 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:39.393 INFO:teuthology.orchestra.run.smithi064.stdout:Total 923 kB/s | 763 kB 00:00 2024-05-08T00:27:39.393 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:27:39.399 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:27:39.399 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:27:39.406 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:27:39.406 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:27:39.478 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:27:39.509 INFO:teuthology.orchestra.run.smithi116.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.5 MB/s | 2.0 MB 00:00 2024-05-08T00:27:39.643 INFO:teuthology.orchestra.run.smithi116.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 377 kB/s | 188 kB 00:00 2024-05-08T00:27:39.646 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:39.646 INFO:teuthology.orchestra.run.smithi116.stdout:Total 3.3 MB/s | 8.6 MB 00:02 2024-05-08T00:27:39.700 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:27:39.718 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:27:39.718 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:27:39.862 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/1 2024-05-08T00:27:39.903 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:27:39.903 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:27:39.973 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/1 2024-05-08T00:27:40.320 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/1 2024-05-08T00:27:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:27:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/1 2024-05-08T00:27:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:40.552 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:27:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:40.552 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:40.552 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:27:40.611 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-08T00:27:40.648 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-08T00:27:40.694 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-08T00:27:40.725 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-08T00:27:40.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-05-08T00:27:40.824 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-08T00:27:40.951 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-08T00:27:41.045 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-08T00:27:41.096 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-08T00:27:41.125 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-08T00:27:41.156 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2529.ge0c1691a.el9 10/12 2024-05-08T00:27:41.353 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:03 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:27:41.481 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 145 k 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-05-08T00:27:41.482 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-05-08T00:27:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 435 k 2024-05-08T00:27:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:27:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.0.0-2529.ge0c169 459 kB/s | 145 kB 00:00 2024-05-08T00:27:41.799 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:41.799 INFO:teuthology.orchestra.run.smithi064.stdout:Total 457 kB/s | 145 kB 00:00 2024-05-08T00:27:41.799 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:27:41.807 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:27:41.807 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:27:41.867 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:27:41.867 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:27:42.025 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:27:42.116 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2529.ge0c1691a. 1/1 2024-05-08T00:27:42.143 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2529.ge0c1691a. 1/1 2024-05-08T00:27:42.143 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:27:42.143 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-08T00:27:42.143 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:42.201 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2529.ge0c1691a.el 11/12 2024-05-08T00:27:42.240 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2529.ge0c1691a.el9.noa 12/12 2024-05-08T00:27:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2529.ge0c1691a. 1/1 2024-05-08T00:27:43.051 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:43.052 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:27:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:27:43.052 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:43.052 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:27:43.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2529.ge0c1691a.el9.noa 12/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2529.ge0c1691a.el 1/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2529.ge0c1691a.el9.noa 2/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2529.ge0c1691a.el9 3/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-08T00:27:43.535 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-08T00:27:43.857 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-08T00:27:43.857 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-08T00:27:43.858 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:43.859 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:27:43.874 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:05 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:27:43.959 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:27:44.004 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:27:44.005 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:27:44.005 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:27:44.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-05-08T00:27:44.119 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-08T00:27:44.665 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:06 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:27:44.685 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:08 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:27:44.793 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:27:44.794 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:44.794 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 3.5 M 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 32 k 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 16 k 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-08T00:27:44.795 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-08T00:27:44.796 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-08T00:27:44.796 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-08T00:27:44.796 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:44.796 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:27:44.796 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:27:44.796 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-05-08T00:27:44.796 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:44.797 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 8.6 M 2024-05-08T00:27:44.797 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 103 M 2024-05-08T00:27:44.797 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:27:44.808 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 7.4 M 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-08T00:27:44.810 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout:Install 12 Packages 2024-05-08T00:27:44.811 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:44.812 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 38 M 2024-05-08T00:27:44.812 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 209 M 2024-05-08T00:27:44.812 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:27:45.044 INFO:teuthology.orchestra.run.smithi116.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 195 kB 00:00 2024-05-08T00:27:45.095 INFO:teuthology.orchestra.run.smithi116.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2024-05-08T00:27:45.120 INFO:teuthology.orchestra.run.smithi116.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-05-08T00:27:45.162 INFO:teuthology.orchestra.run.smithi116.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-05-08T00:27:45.204 INFO:teuthology.orchestra.run.smithi116.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 843 kB/s | 35 kB 00:00 2024-05-08T00:27:45.354 INFO:teuthology.orchestra.run.smithi116.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-05-08T00:27:45.438 INFO:teuthology.orchestra.run.smithi116.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-05-08T00:27:45.597 INFO:teuthology.orchestra.run.smithi116.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-05-08T00:27:45.680 INFO:teuthology.orchestra.run.smithi116.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-05-08T00:27:45.777 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2529.ge0 150 kB/s | 32 kB 00:00 2024-05-08T00:27:45.847 INFO:teuthology.orchestra.run.smithi116.stdout:(10/12): ceph-mgr-diskprediction-local-19.0.0-2 8.1 MB/s | 7.4 MB 00:00 2024-05-08T00:27:45.914 INFO:teuthology.orchestra.run.smithi116.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-05-08T00:27:46.069 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2529.ge0c 33 kB/s | 16 kB 00:00 2024-05-08T00:27:46.177 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): ceph-mgr-dashboard-19.0.0-2529.ge0c1691 5.7 MB/s | 3.5 MB 00:00 2024-05-08T00:27:46.364 INFO:teuthology.orchestra.run.smithi116.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-05-08T00:27:46.365 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:46.365 INFO:teuthology.orchestra.run.smithi116.stdout:Total 24 MB/s | 38 MB 00:01 2024-05-08T00:27:46.619 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-05-08T00:27:46.678 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 442 kB/s | 269 kB 00:00 2024-05-08T00:27:46.717 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:27:46.743 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:27:46.743 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:27:46.811 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-05-08T00:27:46.903 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-05-08T00:27:47.004 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-05-08T00:27:47.070 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 462 kB/s | 31 kB 00:00 2024-05-08T00:27:47.129 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.4 MB/s | 2.0 MB 00:00 2024-05-08T00:27:47.154 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-05-08T00:27:47.180 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 861 kB/s | 863 kB 00:01 2024-05-08T00:27:47.183 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:47.183 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.6 MB/s | 8.6 MB 00:02 2024-05-08T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:27:47.186 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:27:47.238 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:27:47.258 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:27:47.258 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:27:47.448 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:27:47.448 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:27:47.954 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:27:48.020 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-08T00:27:48.062 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-08T00:27:48.069 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:27:48.157 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-08T00:27:48.163 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-08T00:27:48.192 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-08T00:27:48.197 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-08T00:27:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-08T00:27:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-08T00:27:48.358 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-08T00:27:48.377 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-08T00:27:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-08T00:27:48.491 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-08T00:27:48.491 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-08T00:27:48.591 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-08T00:27:48.641 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-08T00:27:48.667 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-08T00:27:48.713 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2529.ge0c1691a.el9 10/12 2024-05-08T00:27:49.154 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-08T00:27:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-08T00:27:49.804 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2529.ge0c1691a.el 11/12 2024-05-08T00:27:49.849 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2529.ge0c1691a.el9.noa 12/12 2024-05-08T00:27:50.496 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-08T00:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2529.ge0c1691a.el9.noa 12/12 2024-05-08T00:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2529.ge0c1691a.el 1/12 2024-05-08T00:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2529.ge0c1691a.el9.noa 2/12 2024-05-08T00:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2529.ge0c1691a.el9 3/12 2024-05-08T00:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-08T00:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-08T00:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-08T00:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-08T00:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-08T00:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-08T00:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-08T00:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-08T00:27:51.408 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-08T00:27:51.408 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:51.408 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-08T00:27:51.409 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-08T00:27:51.410 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:51.410 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:27:51.522 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-08T00:27:51.545 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2529.ge0c16 12/12 2024-05-08T00:27:51.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-08T00:27:52.255 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:14 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:27:52.387 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 7.4 M 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-08T00:27:52.389 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-05-08T00:27:52.390 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-05-08T00:27:52.391 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:27:52.391 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 38 M 2024-05-08T00:27:52.391 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 209 M 2024-05-08T00:27:52.391 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:27:52.871 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-05-08T00:27:52.946 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-05-08T00:27:53.130 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 126 kB/s | 33 kB 00:00 2024-05-08T00:27:53.239 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 168 kB/s | 18 kB 00:00 2024-05-08T00:27:53.414 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-25 11 MB/s | 7.4 MB 00:00 2024-05-08T00:27:53.447 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 169 kB/s | 35 kB 00:00 2024-05-08T00:27:53.706 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 946 kB/s | 244 kB 00:00 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2529.ge0c16 12/12 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2529.ge0c16 1/12 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-08T00:27:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-08T00:27:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-08T00:27:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-08T00:27:54.081 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-05-08T00:27:54.190 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-05-08T00:27:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-08T00:27:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:54.383 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:27:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-08T00:27:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:54.385 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:27:54.657 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.8 MB/s | 4.7 MB 00:01 2024-05-08T00:27:54.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-rook 2024-05-08T00:27:55.199 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.4 MB/s | 5.1 MB 00:01 2024-05-08T00:27:55.378 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:19 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:27:55.509 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:27:55.510 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:55.510 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:27:55.510 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:55.510 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:27:55.510 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 49 k 2024-05-08T00:27:55.510 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:27:55.510 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:55.511 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:27:55.512 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:27:55.512 INFO:teuthology.orchestra.run.smithi116.stdout:Install 13 Packages 2024-05-08T00:27:55.512 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:27:55.512 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.2 M 2024-05-08T00:27:55.512 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 26 M 2024-05-08T00:27:55.512 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:27:56.012 INFO:teuthology.orchestra.run.smithi116.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 93 kB/s | 26 kB 00:00 2024-05-08T00:27:56.028 INFO:teuthology.orchestra.run.smithi116.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 65 kB/s | 19 kB 00:00 2024-05-08T00:27:56.045 INFO:teuthology.orchestra.run.smithi116.stdout:(3/13): ceph-mgr-rook-19.0.0-2529.ge0c1691a.el9 159 kB/s | 49 kB 00:00 2024-05-08T00:27:56.371 INFO:teuthology.orchestra.run.smithi116.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 464 kB/s | 159 kB 00:00 2024-05-08T00:27:56.488 INFO:teuthology.orchestra.run.smithi116.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 484 kB/s | 222 kB 00:00 2024-05-08T00:27:56.505 INFO:teuthology.orchestra.run.smithi116.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 402 kB/s | 54 kB 00:00 2024-05-08T00:27:56.630 INFO:teuthology.orchestra.run.smithi116.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 477 kB/s | 279 kB 00:00 2024-05-08T00:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-05-08T00:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-05-08T00:27:56.973 INFO:teuthology.orchestra.run.smithi116.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 283 kB/s | 59 kB 00:00 2024-05-08T00:27:57.115 INFO:teuthology.orchestra.run.smithi116.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 425 kB/s | 206 kB 00:00 2024-05-08T00:27:57.131 INFO:teuthology.orchestra.run.smithi116.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-05-08T00:27:57.257 INFO:teuthology.orchestra.run.smithi116.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-05-08T00:27:57.257 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:27:57.257 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.3 MB/s | 2.2 MB 00:01 2024-05-08T00:27:57.323 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:27:57.338 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:27:57.338 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:27:57.422 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:27:57.424 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:27:57.931 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:27:58.104 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-08T00:27:58.166 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-08T00:27:58.221 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-08T00:27:58.259 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-08T00:27:58.291 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-08T00:27:58.375 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-08T00:27:58.486 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-08T00:27:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-08T00:27:59.244 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-08T00:27:59.290 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-08T00:27:59.324 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-08T00:27:59.372 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-08T00:27:59.394 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-2529.ge0c1691a.el9.noarch 13/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2529.ge0c1691a.el9.noarch 13/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2529.ge0c1691a.el9.noarch 1/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-08T00:28:00.183 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-08T00:28:00.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-08T00:28:00.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-08T00:28:00.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-08T00:28:00.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-08T00:28:00.497 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:00.497 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-08T00:28:00.498 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-08T00:28:00.499 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:00.499 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:00.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-cephadm 2024-05-08T00:28:01.314 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:25 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:01.437 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:01.438 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:28:01.438 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:01.438 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:28:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 142 k 2024-05-08T00:28:01.438 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:28:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-08T00:28:01.438 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-08T00:28:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-08T00:28:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-08T00:28:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-08T00:28:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:01.439 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:28:01.439 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:01.439 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2024-05-08T00:28:01.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:01.440 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 7.0 M 2024-05-08T00:28:01.440 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 33 M 2024-05-08T00:28:01.440 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-05-08T00:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2529.ge0c1691a.e 367 kB/s | 142 kB 00:00 2024-05-08T00:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-05-08T00:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 239 kB/s | 58 kB 00:00 2024-05-08T00:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 875 kB/s | 51 kB 00:00 2024-05-08T00:28:03.426 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.3 MB/s | 6.0 MB 00:01 2024-05-08T00:28:03.429 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:03.429 INFO:teuthology.orchestra.run.smithi116.stdout:Total 3.5 MB/s | 7.0 MB 00:01 2024-05-08T00:28:03.501 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:28:03.512 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:28:03.512 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:28:03.609 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:28:03.610 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:28:03.914 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:28:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-08T00:28:04.145 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-08T00:28:04.655 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-08T00:28:04.738 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-08T00:28:04.810 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-08T00:28:04.840 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 6/6 2024-05-08T00:28:05.643 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 6/6 2024-05-08T00:28:05.644 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/6 2024-05-08T00:28:05.644 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-08T00:28:05.644 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-08T00:28:05.644 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-08T00:28:05.644 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-08T00:28:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-08T00:28:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:05.906 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:28:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:28:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-08T00:28:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-08T00:28:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-08T00:28:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-08T00:28:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-08T00:28:05.907 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:05.907 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:06.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-fuse 2024-05-08T00:28:06.747 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:30 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:06.871 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:06.872 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:06.872 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:28:06.872 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 890 k 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout:Install 2 Packages 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 970 k 2024-05-08T00:28:06.873 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 2.8 M 2024-05-08T00:28:06.874 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:28:07.037 INFO:teuthology.orchestra.run.smithi116.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 677 kB/s | 80 kB 00:00 2024-05-08T00:28:07.395 INFO:teuthology.orchestra.run.smithi116.stdout:(2/2): ceph-fuse-19.0.0-2529.ge0c1691a.el9.x86_ 1.8 MB/s | 890 kB 00:00 2024-05-08T00:28:07.398 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:07.398 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.8 MB/s | 970 kB 00:00 2024-05-08T00:28:07.409 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:28:07.432 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:28:07.432 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:28:07.510 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:28:07.511 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:28:07.699 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:28:07.840 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-08T00:28:07.919 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/2 2024-05-08T00:28:08.570 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/2 2024-05-08T00:28:08.570 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/2 2024-05-08T00:28:08.870 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-08T00:28:08.870 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:08.870 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:28:08.870 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-08T00:28:08.870 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:08.870 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:09.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-volume 2024-05-08T00:28:09.732 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:33 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:09.857 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:09.858 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:09.858 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-05-08T00:28:09.858 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:09.858 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 264 k 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout:Install 2 Packages 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 341 k 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 1.5 M 2024-05-08T00:28:09.859 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:28:10.009 INFO:teuthology.orchestra.run.smithi116.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 898 kB/s | 77 kB 00:00 2024-05-08T00:28:10.184 INFO:teuthology.orchestra.run.smithi116.stdout:(2/2): ceph-volume-19.0.0-2529.ge0c1691a.el9.no 1.0 MB/s | 264 kB 00:00 2024-05-08T00:28:10.185 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:10.185 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.0 MB/s | 341 kB 00:00 2024-05-08T00:28:10.197 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:28:10.206 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:28:10.206 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:28:10.241 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:28:10.241 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:28:10.383 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:28:10.568 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-08T00:28:10.611 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-2529.ge0c1691a.el9.noarch 2/2 2024-05-08T00:28:10.627 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-2529.ge0c1691a.el9.noarch 2/2 2024-05-08T00:28:10.627 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:28:10.627 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-08T00:28:10.627 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:11.316 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-2529.ge0c1691a.el9.noarch 1/2 2024-05-08T00:28:11.604 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-08T00:28:11.605 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:11.605 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:28:11.605 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:28:11.605 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-08T00:28:11.605 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:11.605 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:11.918 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados-devel 2024-05-08T00:28:12.502 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:36 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:12.628 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:12.629 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:12.629 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-05-08T00:28:12.629 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:12.629 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:28:12.629 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 127 k 2024-05-08T00:28:12.629 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:28:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-05-08T00:28:12.630 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 127 k 2024-05-08T00:28:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 456 k 2024-05-08T00:28:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:28:12.866 INFO:teuthology.orchestra.run.smithi116.stdout:librados-devel-19.0.0-2529.ge0c1691a.el9.x86_64 539 kB/s | 127 kB 00:00 2024-05-08T00:28:12.866 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:12.866 INFO:teuthology.orchestra.run.smithi116.stdout:Total 535 kB/s | 127 kB 00:00 2024-05-08T00:28:12.866 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:28:12.872 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:28:12.872 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:28:12.940 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:28:12.940 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:28:13.094 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:28:13.196 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:13.505 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 862 kB/s | 16 MB 00:19 2024-05-08T00:28:13.506 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:13.507 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.8 MB/s | 38 MB 00:21 2024-05-08T00:28:13.630 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:13.878 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:13.879 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:28:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:13.879 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:13.906 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:13.906 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:14.142 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs2 2024-05-08T00:28:14.362 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:14.363 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:14.729 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:38 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:14.811 INFO:teuthology.orchestra.run.smithi116.stdout:Package libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:14.855 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:14.856 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:28:14.856 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:14.919 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs-devel 2024-05-08T00:28:15.146 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:15.216 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-08T00:28:15.254 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-08T00:28:15.360 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-08T00:28:15.397 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-08T00:28:15.488 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:39 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-08T00:28:15.610 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:15.610 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-05-08T00:28:15.610 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:15.610 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:28:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 32 k 2024-05-08T00:28:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:15.611 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:28:15.611 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:15.611 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-05-08T00:28:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:15.611 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 32 k 2024-05-08T00:28:15.611 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 142 k 2024-05-08T00:28:15.611 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:28:15.612 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-08T00:28:15.700 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-08T00:28:15.768 INFO:teuthology.orchestra.run.smithi116.stdout:libcephfs-devel-19.0.0-2529.ge0c1691a.el9.x86_6 202 kB/s | 32 kB 00:00 2024-05-08T00:28:15.769 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:15.769 INFO:teuthology.orchestra.run.smithi116.stdout:Total 200 kB/s | 32 kB 00:00 2024-05-08T00:28:15.769 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:28:15.773 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:28:15.773 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:28:15.784 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:28:15.784 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:28:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:28:15.974 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:16.352 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:16.405 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-08T00:28:16.525 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-08T00:28:16.586 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:16.586 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:16.586 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:28:16.586 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:16.586 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:16.586 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:16.752 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados2 2024-05-08T00:28:17.319 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:41 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:17.398 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:17.439 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:17.440 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:28:17.440 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:17.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librbd1 2024-05-08T00:28:17.849 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-08T00:28:18.073 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:42 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:18.154 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:18.195 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:18.196 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:28:18.196 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:18.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rados 2024-05-08T00:28:18.836 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:42 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:18.917 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rados-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:18.937 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-08T00:28:18.958 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:18.959 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:28:18.959 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2529.ge0c16 12/12 2024-05-08T00:28:19.026 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rgw 2024-05-08T00:28:19.593 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:43 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:19.715 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:19.716 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:28:19.716 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:19.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-cephfs 2024-05-08T00:28:20.351 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:44 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:20.432 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-cephfs-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:20.475 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:20.476 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:28:20.476 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:20.543 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rbd 2024-05-08T00:28:21.117 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:45 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:21.197 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:21.240 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:21.241 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:28:21.241 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:21.309 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-fuse 2024-05-08T00:28:21.566 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2529.ge0c16 12/12 2024-05-08T00:28:21.566 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2529.ge0c16 1/12 2024-05-08T00:28:21.566 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-08T00:28:21.567 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-08T00:28:21.567 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-08T00:28:21.567 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-08T00:28:21.567 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-08T00:28:21.567 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-08T00:28:21.567 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-08T00:28:21.567 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-08T00:28:21.567 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-08T00:28:21.567 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-08T00:28:21.877 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:45 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:21.957 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-08T00:28:21.957 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:21.957 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:21.957 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:21.958 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:21.997 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:21.998 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:21.998 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-05-08T00:28:21.998 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:21.998 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:28:21.998 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 85 k 2024-05-08T00:28:21.998 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:21.999 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:28:21.999 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:21.999 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-05-08T00:28:21.999 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:21.999 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 85 k 2024-05-08T00:28:21.999 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 231 k 2024-05-08T00:28:21.999 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:28:22.214 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-fuse-19.0.0-2529.ge0c1691a.el9.x86_64.rpm 398 kB/s | 85 kB 00:00 2024-05-08T00:28:22.215 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:22.215 INFO:teuthology.orchestra.run.smithi116.stdout:Total 395 kB/s | 85 kB 00:00 2024-05-08T00:28:22.215 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:28:22.221 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:28:22.221 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:28:22.284 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:28:22.285 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:28:22.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-05-08T00:28:22.427 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:28:22.528 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:22.891 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:22.927 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:23.059 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:23.060 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:23.060 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:28:23.060 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:23.060 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:28:23.060 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 49 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-08T00:28:23.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-08T00:28:23.062 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:23.062 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:28:23.062 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:23.062 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-05-08T00:28:23.062 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:23.062 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.2 M 2024-05-08T00:28:23.063 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-05-08T00:28:23.063 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:28:23.127 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:23.128 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:23.128 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:28:23.128 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:23.128 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:23.128 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:23.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-mirror 2024-05-08T00:28:23.463 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 366 kB/s | 26 kB 00:00 2024-05-08T00:28:23.489 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 202 kB/s | 19 kB 00:00 2024-05-08T00:28:23.555 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.4 MB/s | 159 kB 00:00 2024-05-08T00:28:23.581 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-05-08T00:28:23.606 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): ceph-mgr-rook-19.0.0-2529.ge0c1691a.el9 232 kB/s | 49 kB 00:00 2024-05-08T00:28:23.656 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 712 kB/s | 54 kB 00:00 2024-05-08T00:28:23.707 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-05-08T00:28:23.882 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-05-08T00:28:23.907 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2024-05-08T00:28:23.927 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:47 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:24.052 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:24.053 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:24.053 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-05-08T00:28:24.053 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:24.053 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:28:24.053 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 3.1 M 2024-05-08T00:28:24.054 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:24.054 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:28:24.054 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:24.054 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-05-08T00:28:24.054 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:24.054 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 3.1 M 2024-05-08T00:28:24.054 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 13 M 2024-05-08T00:28:24.054 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:28:24.066 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 573 kB/s | 206 kB 00:00 2024-05-08T00:28:24.124 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 272 kB/s | 59 kB 00:00 2024-05-08T00:28:24.249 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-05-08T00:28:24.274 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2024-05-08T00:28:24.278 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:24.278 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-05-08T00:28:24.345 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:24.360 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:24.360 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:24.441 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-mirror-19.0.0-2529.ge0c1691a.el9.x86_64.rpm 8.0 MB/s | 3.1 MB 00:00 2024-05-08T00:28:24.442 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:24.442 INFO:teuthology.orchestra.run.smithi116.stdout:Total 7.9 MB/s | 3.1 MB 00:00 2024-05-08T00:28:24.443 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:28:24.451 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:28:24.451 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:28:24.453 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:24.454 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:24.532 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:28:24.532 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:28:24.960 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:28:24.982 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:25.009 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:25.033 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:25.033 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:28:25.033 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-08T00:28:25.033 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-08T00:28:25.033 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-08T00:28:25.034 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-08T00:28:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-08T00:28:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-08T00:28:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-08T00:28:25.470 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-08T00:28:25.567 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-08T00:28:25.678 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-08T00:28:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-08T00:28:26.174 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:26.175 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:26.175 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:28:26.175 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:26.175 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:26.175 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:26.413 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-nbd 2024-05-08T00:28:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-08T00:28:26.543 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-08T00:28:26.576 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-08T00:28:26.625 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-08T00:28:26.654 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-2529.ge0c1691a.el9.noarch 13/13 2024-05-08T00:28:27.008 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:51 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:27.133 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:27.133 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 170 k 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 170 k 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 494 k 2024-05-08T00:28:27.134 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-05-08T00:28:27.362 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-nbd-19.0.0-2529.ge0c1691a.el9.x86_64.rpm 749 kB/s | 170 kB 00:00 2024-05-08T00:28:27.363 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:27.363 INFO:teuthology.orchestra.run.smithi116.stdout:Total 744 kB/s | 170 kB 00:00 2024-05-08T00:28:27.363 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-05-08T00:28:27.371 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-05-08T00:28:27.371 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-05-08T00:28:27.391 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2529.ge0c1691a.el9.noarch 13/13 2024-05-08T00:28:27.391 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2529.ge0c1691a.el9.noarch 1/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-08T00:28:27.392 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-08T00:28:27.434 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-05-08T00:28:27.434 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-05-08T00:28:27.583 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-05-08T00:28:27.676 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-08T00:28:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:27.715 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:28:27.715 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:27.716 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:27.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-05-08T00:28:28.081 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:28.309 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:28.309 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:28.309 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-05-08T00:28:28.309 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:28.309 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:28:28.309 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:28.537 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-mirror 2024-05-08T00:28:28.594 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:50 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:28.720 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:28.721 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:28.721 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:28:28.721 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:28.721 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 142 k 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-05-08T00:28:28.722 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:28.723 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 7.0 M 2024-05-08T00:28:28.723 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-05-08T00:28:28.723 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:28:29.117 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:53 ago on Wed 08 May 2024 12:26:36 AM UTC. 2024-05-08T00:28:29.198 INFO:teuthology.orchestra.run.smithi116.stdout:Package rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:29.242 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-05-08T00:28:29.243 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-05-08T00:28:29.243 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-05-08T00:28:29.276 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 730 kB/s | 248 kB 00:00 2024-05-08T00:28:29.325 DEBUG:teuthology.parallel:result is None 2024-05-08T00:28:29.360 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2529.ge0c1691a.e 335 kB/s | 142 kB 00:00 2024-05-08T00:28:29.569 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 277 kB/s | 58 kB 00:00 2024-05-08T00:28:29.594 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-05-08T00:28:29.619 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-05-08T00:28:30.486 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.8 MB/s | 6.0 MB 00:01 2024-05-08T00:28:30.488 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:30.489 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.0 MB/s | 7.0 MB 00:01 2024-05-08T00:28:30.562 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:30.574 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:30.574 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:30.677 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:30.678 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:30.990 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:31.161 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-08T00:28:31.214 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-08T00:28:31.769 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-08T00:28:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-08T00:28:31.939 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-08T00:28:31.966 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 6/6 2024-05-08T00:28:32.873 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 6/6 2024-05-08T00:28:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 1/6 2024-05-08T00:28:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-08T00:28:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-08T00:28:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-08T00:28:32.874 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-08T00:28:33.138 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-08T00:28:33.138 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:33.138 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:33.138 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:28:33.138 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-08T00:28:33.139 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-08T00:28:33.139 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-08T00:28:33.139 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-08T00:28:33.139 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-08T00:28:33.139 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:33.139 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:33.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-05-08T00:28:34.021 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:56 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:34.151 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 890 k 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-05-08T00:28:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:34.153 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 970 k 2024-05-08T00:28:34.153 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.8 M 2024-05-08T00:28:34.153 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:28:34.571 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 557 kB/s | 80 kB 00:00 2024-05-08T00:28:34.855 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.0.0-2529.ge0c1691a.el9.x86_ 2.0 MB/s | 890 kB 00:00 2024-05-08T00:28:34.855 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:34.855 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.3 MB/s | 970 kB 00:00 2024-05-08T00:28:34.868 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:34.892 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:34.892 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:34.974 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:34.975 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:35.166 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:35.294 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-08T00:28:35.374 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/2 2024-05-08T00:28:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 2/2 2024-05-08T00:28:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/2 2024-05-08T00:28:36.366 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-08T00:28:36.366 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:36.366 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:36.366 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-08T00:28:36.367 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:36.367 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:36.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-05-08T00:28:37.223 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:59 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:37.353 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-2529.ge0c1691a.el9 ceph-noarch 264 k 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-05-08T00:28:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:37.355 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 341 k 2024-05-08T00:28:37.355 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.5 M 2024-05-08T00:28:37.355 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:28:37.818 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): ceph-volume-19.0.0-2529.ge0c1691a.el9.no 768 kB/s | 264 kB 00:00 2024-05-08T00:28:38.176 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 110 kB/s | 77 kB 00:00 2024-05-08T00:28:38.177 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:38.177 INFO:teuthology.orchestra.run.smithi064.stdout:Total 414 kB/s | 341 kB 00:00 2024-05-08T00:28:38.190 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:38.201 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:38.201 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:38.235 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:38.235 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:38.385 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-08T00:28:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-2529.ge0c1691a.el9.noarch 2/2 2024-05-08T00:28:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-2529.ge0c1691a.el9.noarch 2/2 2024-05-08T00:28:38.620 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:28:38.620 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-08T00:28:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:39.302 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-2529.ge0c1691a.el9.noarch 1/2 2024-05-08T00:28:39.580 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-08T00:28:39.580 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:39.580 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:39.580 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-2529.ge0c1691a.el9.noarch 2024-05-08T00:28:39.580 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-08T00:28:39.580 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:39.580 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:39.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-05-08T00:28:40.486 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:02 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:40.615 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:40.616 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:40.616 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-05-08T00:28:40.616 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 127 k 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 127 k 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-05-08T00:28:40.617 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:28:40.938 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.0.0-2529.ge0c1691a.el9.x86_64 394 kB/s | 127 kB 00:00 2024-05-08T00:28:40.939 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:40.939 INFO:teuthology.orchestra.run.smithi064.stdout:Total 392 kB/s | 127 kB 00:00 2024-05-08T00:28:40.939 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:40.945 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:40.945 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:41.018 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:41.018 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:41.177 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:41.282 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:41.695 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:41.970 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:41.970 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:41.970 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:41.970 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:41.971 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:41.971 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:42.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-05-08T00:28:42.834 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:04 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:42.919 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:42.964 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:42.965 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:28:42.965 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:43.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-05-08T00:28:43.620 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:05 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:43.749 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:43.750 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:43.750 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 32 k 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 32 k 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 142 k 2024-05-08T00:28:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:28:43.925 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.0.0-2529.ge0c1691a.el9.x86_6 183 kB/s | 32 kB 00:00 2024-05-08T00:28:43.925 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:43.925 INFO:teuthology.orchestra.run.smithi064.stdout:Total 182 kB/s | 32 kB 00:00 2024-05-08T00:28:43.925 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:43.929 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:43.929 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:43.941 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:43.942 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:44.124 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:44.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:44.702 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:44.703 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:44.703 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:44.703 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:44.703 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:44.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-05-08T00:28:45.501 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:07 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:45.585 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:45.628 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:45.630 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:28:45.630 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:45.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-05-08T00:28:46.313 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:08 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:46.397 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:46.440 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:46.441 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:28:46.441 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:46.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-05-08T00:28:47.115 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:09 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:47.199 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:47.243 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:47.244 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:28:47.244 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:47.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-05-08T00:28:47.910 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:09 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:47.995 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:48.038 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:48.039 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:28:48.039 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:48.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-05-08T00:28:48.703 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:10 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:48.788 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:48.833 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:48.834 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:28:48.834 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:48.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-05-08T00:28:49.505 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:11 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:49.588 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:49.633 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:49.634 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:28:49.634 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:49.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-05-08T00:28:50.298 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:12 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:50.425 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:50.426 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:50.426 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-05-08T00:28:50.426 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:50.426 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:28:50.426 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 85 k 2024-05-08T00:28:50.426 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:50.427 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:28:50.427 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:50.427 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-05-08T00:28:50.427 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:50.427 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-05-08T00:28:50.427 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-05-08T00:28:50.427 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.0.0-2529.ge0c1691a.el9.x86_64.rpm 291 kB/s | 85 kB 00:00 2024-05-08T00:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout:Total 290 kB/s | 85 kB 00:00 2024-05-08T00:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:50.796 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:50.796 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:50.942 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:51.546 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:51.783 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:51.783 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:51.783 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:51.784 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:51.784 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:51.784 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:52.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-05-08T00:28:52.603 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:14 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:52.733 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:52.733 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 3.1 M 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.1 M 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 13 M 2024-05-08T00:28:52.734 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:28:53.115 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.0.0-2529.ge0c1691a.el9.x86_64.rpm 8.1 MB/s | 3.1 MB 00:00 2024-05-08T00:28:53.116 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:53.116 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.1 MB/s | 3.1 MB 00:00 2024-05-08T00:28:53.117 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:53.126 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:53.127 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:53.212 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:53.213 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:53.671 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:53.733 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:53.760 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:53.760 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-08T00:28:53.760 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-08T00:28:53.760 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-08T00:28:53.760 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-08T00:28:53.761 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:54.810 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:54.811 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:54.811 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:55.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-05-08T00:28:55.754 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:17 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:55.880 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-2529.ge0c1691a.el9 ceph 170 k 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-05-08T00:28:55.881 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:55.882 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 170 k 2024-05-08T00:28:55.882 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 494 k 2024-05-08T00:28:55.882 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-05-08T00:28:56.236 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.0.0-2529.ge0c1691a.el9.x86_64.rpm 482 kB/s | 170 kB 00:00 2024-05-08T00:28:56.236 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-05-08T00:28:56.236 INFO:teuthology.orchestra.run.smithi064.stdout:Total 480 kB/s | 170 kB 00:00 2024-05-08T00:28:56.237 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-05-08T00:28:56.245 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-05-08T00:28:56.245 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-05-08T00:28:56.312 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-05-08T00:28:56.312 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-05-08T00:28:56.476 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-05-08T00:28:56.578 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:57.084 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 1/1 2024-05-08T00:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-05-08T00:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-2529.ge0c1691a.el9.x86_64 2024-05-08T00:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:57.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-05-08T00:28:58.173 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:20 ago on Wed 08 May 2024 12:26:38 AM UTC. 2024-05-08T00:28:58.259 INFO:teuthology.orchestra.run.smithi064.stdout:Package rbd-mirror-2:19.0.0-2529.ge0c1691a.el9.x86_64 is already installed. 2024-05-08T00:28:58.303 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-05-08T00:28:58.304 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-05-08T00:28:58.304 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-05-08T00:28:58.381 DEBUG:teuthology.parallel:result is None 2024-05-08T00:28:58.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e0c1691aade07c0b78ccff9531f65d3c80e3e220 2024-05-08T00:28:58.604 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-08T00:28:58.626 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-2529.ge0c1691a.el9 2024-05-08T00:28:58.627 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2529.ge0c1691a.el9 2024-05-08T00:28:58.627 INFO:teuthology.task.install:The correct ceph version 19.0.0-2529.ge0c1691a is installed. 2024-05-08T00:28:58.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e0c1691aade07c0b78ccff9531f65d3c80e3e220 2024-05-08T00:28:58.770 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-08T00:28:58.796 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-2529.ge0c1691a.el9 2024-05-08T00:28:58.796 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2529.ge0c1691a.el9 2024-05-08T00:28:58.796 INFO:teuthology.task.install:The correct ceph version 19.0.0-2529.ge0c1691a is installed. 2024-05-08T00:28:58.798 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-08T00:28:58.799 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:28:58.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-08T00:28:58.825 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:28:58.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-08T00:28:58.866 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-08T00:28:58.866 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:28:58.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-05-08T00:28:58.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-08T00:28:58.957 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:28:58.957 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-05-08T00:28:58.985 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-08T00:28:59.052 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-08T00:28:59.053 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:28:59.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-08T00:28:59.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-08T00:28:59.144 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:28:59.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-08T00:28:59.171 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-08T00:28:59.237 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-08T00:28:59.238 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:28:59.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-05-08T00:28:59.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-08T00:28:59.330 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:28:59.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-05-08T00:28:59.356 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-08T00:28:59.421 INFO:teuthology.run_tasks:Running task ceph... 2024-05-08T00:28:59.516 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-08T00:28:59.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-05-08T00:28:59.519 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2024-05-08T00:28:59.543 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-08T00:28:59.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-08T00:28:59.585 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-08T00:28:59.621 INFO:tasks.ceph:Creating extra log directories... 2024-05-08T00:28:59.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-08T00:28:59.651 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-08T00:28:59.700 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-05-08T00:28:59.701 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-08T00:28:59.701 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630', 'branch': 'wip-dis-testing-squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7696630', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-squid', 'sha1': '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5306, 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220', 'sleep_before_teardown': 0, 'subset': '21/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNse/AFoy1yDcGyY+MS/1v1sY6rGKOdpmwkIA7spCDsdeG/eq/LWdPvBzKpmx6d2QajiJpoDqcDNFeS5zRkvOOI=', 'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJY4Atyn5pq0iMFwyGZY2SznIWkQuGBB2prrTIQN9ZryfhoD6Ci81KzAnIOQdzmtaHC2/SgzepfCshe0z9Vzcbc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-07_23:02:08', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-08T00:28:59.701 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-05-08T00:28:59.717 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-05-08T00:28:59.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-05-08T00:28:59.773 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-05-08T00:28:59.807 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:28:59.807 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-05-08T00:28:59.854 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-08T00:28:59.854 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-05-08T00:28:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-08T00:28:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T00:28:59.910 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-08T00:28:59.910 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T00:28:59.910 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T00:28:59.910 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-05-08 00:28:54.019502842 +0000 2024-05-08T00:28:59.910 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-05-08 00:25:55.410864245 +0000 2024-05-08T00:28:59.910 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-05-08 00:25:55.410864245 +0000 2024-05-08T00:28:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-05-08 00:25:55.410864245 +0000 2024-05-08T00:28:59.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-08T00:28:59.974 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-05-08T00:28:59.974 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-05-08T00:28:59.974 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000260266 s, 2.0 MB/s 2024-05-08T00:28:59.975 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-08T00:29:00.032 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-05-08T00:29:00.088 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-08T00:29:00.089 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T00:29:00.089 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-08T00:29:00.089 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T00:29:00.089 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T00:29:00.089 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-05-08 00:28:54.019502842 +0000 2024-05-08T00:29:00.089 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-05-08 00:25:55.758856971 +0000 2024-05-08T00:29:00.089 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-05-08 00:25:55.758856971 +0000 2024-05-08T00:29:00.089 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-05-08 00:25:55.758856971 +0000 2024-05-08T00:29:00.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-08T00:29:00.152 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-05-08T00:29:00.153 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-05-08T00:29:00.153 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000181066 s, 2.8 MB/s 2024-05-08T00:29:00.154 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-08T00:29:00.210 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-05-08T00:29:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-08T00:29:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T00:29:00.267 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-08T00:29:00.267 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T00:29:00.267 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T00:29:00.267 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-05-08 00:28:54.020502822 +0000 2024-05-08T00:29:00.267 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-05-08 00:25:56.114849529 +0000 2024-05-08T00:29:00.267 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-05-08 00:25:56.114849529 +0000 2024-05-08T00:29:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-05-08 00:25:56.114849529 +0000 2024-05-08T00:29:00.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-08T00:29:00.330 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-05-08T00:29:00.330 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-05-08T00:29:00.330 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000188533 s, 2.7 MB/s 2024-05-08T00:29:00.331 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-08T00:29:00.388 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-05-08T00:29:00.445 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-08T00:29:00.445 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T00:29:00.445 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-08T00:29:00.445 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T00:29:00.445 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T00:29:00.445 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-05-08 00:28:54.020502822 +0000 2024-05-08T00:29:00.446 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-05-08 00:25:56.481841857 +0000 2024-05-08T00:29:00.446 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-05-08 00:25:56.481841857 +0000 2024-05-08T00:29:00.446 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-05-08 00:25:56.481841857 +0000 2024-05-08T00:29:00.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-08T00:29:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-05-08T00:29:00.508 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-05-08T00:29:00.509 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000216615 s, 2.4 MB/s 2024-05-08T00:29:00.510 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-08T00:29:00.566 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-08T00:29:00.566 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-08T00:29:00.566 INFO:tasks.ceph:Generating config... 2024-05-08T00:29:00.567 INFO:tasks.ceph:[client] debug rbd = 20 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client] log to stderr = False 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client] rbd default features = 125 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:00.568 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:00.569 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:00.569 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:00.569 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:00.569 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-08T00:29:00.569 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-05-08T00:29:00.569 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-08T00:29:00.569 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-08T00:29:00.569 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-08T00:29:00.569 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-08T00:29:00.569 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-08T00:29:00.569 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-08T00:29:00.569 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-08T00:29:00.569 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-05-08T00:29:00.569 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-05-08T00:29:00.569 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-08T00:29:00.569 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-08T00:29:00.569 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-08T00:29:00.570 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-08T00:29:00.570 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-08T00:29:00.570 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-08T00:29:00.570 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-08T00:29:00.570 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-08T00:29:00.570 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-08T00:29:00.570 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-08T00:29:00.570 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-08T00:29:00.570 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-05-08T00:29:00.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-05-08T00:29:00.657 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/cluster1.keyring 2024-05-08T00:29:00.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-05-08T00:29:00.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-05-08T00:29:00.769 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.64')] 2024-05-08T00:29:00.770 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.64', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-05-08T00:29:00.770 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:00.770 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-08T00:29:00.829 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.64 --print /home/ubuntu/cephtest/cluster1.monmap 2024-05-08T00:29:00.909 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid 4f2229d1-1d9b-4660-ad3b-ce8a6d783878 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 4f2229d1-1d9b-4660-ad3b-ce8a6d783878 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2024-05-08T00:29:00.911402+0000 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-05-08T00:29:00.911402+0000 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 17 (quincy) 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 1 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-05-08T00:29:00.910 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-05-08T00:29:00.912 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-08T00:29:00.967 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 4f2229d1-1d9b-4660-ad3b-ce8a6d783878... 2024-05-08T00:29:00.968 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-05-08T00:29:01.010 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-05-08T00:29:01.052 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-05-08T00:29:01.052 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.053 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-05-08T00:29:01.054 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = 172.21.15.64 2024-05-08T00:29:01.055 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on pool no app = False 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 4f2229d1-1d9b-4660-ad3b-ce8a6d783878 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-05-08T00:29:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-05-08T00:29:01.057 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = snappy 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 1/20 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 1/20 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 4/10 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-05-08T00:29:01.058 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-05-08T00:29:01.059 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T00:29:01.060 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T00:29:01.061 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: debug rbd = 20 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: debug rbd_mirror = 15 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: log to stderr = False 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: rbd default features = 125 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.0] 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.1] 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.2] 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.062 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.3] 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.4] 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.5] 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.6] 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.063 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-05-08T00:29:01.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-05-08T00:29:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-05-08T00:29:01.066 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = 172.21.15.64 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon client directed command retry = 5 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on pool no app = False 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject socket failures = 5000 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 4f2229d1-1d9b-4660-ad3b-ce8a6d783878 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-05-08T00:29:01.067 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.068 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore compression algorithm = snappy 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore compression mode = aggressive 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-05-08T00:29:01.069 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-05-08T00:29:01.070 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-05-08T00:29:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-05-08T00:29:01.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: debug rbd = 20 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: debug rbd_mirror = 15 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: log to stderr = False 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: rbd default features = 125 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.0] 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.1] 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.2] 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.3] 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.4] 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.5] 2024-05-08T00:29:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.075 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.6] 2024-05-08T00:29:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:01.076 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-05-08T00:29:01.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-05-08T00:29:01.172 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-08T00:29:01.172 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:01.172 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-05-08T00:29:01.189 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:01.189 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-05-08T00:29:01.244 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T00:29:01.245 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:01.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/cluster1.keyring 2024-05-08T00:29:01.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-05-08T00:29:01.320 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:01.321 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-05-08T00:29:01.377 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T00:29:01.377 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:01.377 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/cluster1.keyring 2024-05-08T00:29:01.377 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-05-08T00:29:01.412 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:01.412 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-05-08T00:29:01.468 INFO:tasks.ceph:Setting up mon nodes... 2024-05-08T00:29:01.468 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-08T00:29:01.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-05-08T00:29:01.520 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-05-08T00:29:01.523 INFO:tasks.ceph:Setting up mds nodes... 2024-05-08T00:29:01.523 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-08T00:29:01.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-05-08T00:29:01.566 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-05-08T00:29:01.579 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-05-08T00:29:01.624 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-05-08T00:29:01.638 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-05-08T00:29:01.678 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-05-08T00:29:01.691 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-05-08T00:29:01.771 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-05-08T00:29:01.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-05-08T00:29:01.864 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-05-08T00:29:01.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-05-08T00:29:01.957 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-05-08T00:29:01.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-05-08T00:29:02.051 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-05-08T00:29:02.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-05-08T00:29:02.145 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-05-08T00:29:02.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-05-08T00:29:02.238 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-05-08T00:29:02.251 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-08T00:29:02.251 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-08T00:29:02.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-05-08T00:29:02.280 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-08T00:29:02.280 INFO:tasks.ceph:role: cluster1.osd.0 2024-05-08T00:29:02.280 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T00:29:02.280 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-08T00:29:02.355 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T00:29:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T00:29:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T00:29:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-08T00:29:02.355 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T00:29:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-05-08T00:29:02.355 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T00:29:02.355 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-08T00:29:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T00:29:02.356 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T00:29:02.879 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-05-08T00:29:02.881 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-05-08T00:29:02.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-05-08T00:29:03.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-05-08T00:29:03.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-05-08T00:29:03.134 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-08T00:29:03.134 INFO:tasks.ceph:role: cluster1.osd.1 2024-05-08T00:29:03.134 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T00:29:03.134 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-08T00:29:03.210 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T00:29:03.210 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T00:29:03.210 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T00:29:03.210 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-08T00:29:03.210 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T00:29:03.210 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-05-08T00:29:03.211 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T00:29:03.211 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-08T00:29:03.211 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T00:29:03.211 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T00:29:03.658 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-05-08T00:29:03.661 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-05-08T00:29:03.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-05-08T00:29:03.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-05-08T00:29:03.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-05-08T00:29:03.858 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-08T00:29:03.858 INFO:tasks.ceph:role: cluster1.osd.2 2024-05-08T00:29:03.858 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T00:29:03.858 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-08T00:29:03.934 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T00:29:03.934 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T00:29:03.934 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T00:29:03.934 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-08T00:29:03.934 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T00:29:03.934 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-05-08T00:29:03.934 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T00:29:03.934 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-08T00:29:03.935 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T00:29:03.935 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T00:29:04.424 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-05-08T00:29:04.426 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-05-08T00:29:04.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-05-08T00:29:04.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-05-08T00:29:04.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-05-08T00:29:04.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:04.695+0000 7fbce7826640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-05-08T00:29:04.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:04.696+0000 7fbce7826640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-05-08T00:29:04.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:04.696+0000 7fbce7826640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-05-08T00:29:04.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:04.696+0000 7fbce7826640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-05-08T00:29:04.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:04.696+0000 7fbce7826640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-05-08T00:29:04.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:04.700+0000 7fbce7826640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-05-08T00:29:07.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-05-08T00:29:07.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-05-08T00:29:07.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:07.675+0000 7f63f800f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-05-08T00:29:07.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:07.676+0000 7f63f800f640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-05-08T00:29:07.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:07.676+0000 7f63f800f640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-05-08T00:29:07.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:07.676+0000 7f63f800f640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-05-08T00:29:07.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:07.676+0000 7f63f800f640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-05-08T00:29:07.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:07.676+0000 7f63f800f640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-05-08T00:29:10.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-05-08T00:29:10.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-05-08T00:29:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:10.656+0000 7f2ebb412640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-05-08T00:29:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:10.656+0000 7f2ebb412640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-05-08T00:29:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:10.656+0000 7f2ebb412640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-05-08T00:29:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:10.656+0000 7f2ebb412640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-05-08T00:29:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:10.656+0000 7f2ebb412640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-05-08T00:29:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-05-08T00:29:10.657+0000 7f2ebb412640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-05-08T00:29:13.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-05-08T00:29:13.540 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-08T00:29:13.540 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:13.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-05-08T00:29:13.608 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:13.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-05-08T00:29:13.674 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:13.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-05-08T00:29:13.741 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:13.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-05-08T00:29:13.808 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:13.808 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-05-08T00:29:13.864 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:13.864 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-05-08T00:29:13.884 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:13.884 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-05-08T00:29:13.942 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:13.942 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-05-08T00:29:14.000 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:14.000 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-05-08T00:29:14.058 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:14.058 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-05-08T00:29:14.117 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:14.117 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-05-08T00:29:14.174 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:14.175 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-05-08T00:29:14.232 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:14.232 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-05-08T00:29:14.289 INFO:tasks.ceph:Adding keys to all mons... 2024-05-08T00:29:14.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/cluster1.keyring 2024-05-08T00:29:14.315 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-05-08T00:29:14.315 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBNxzpmbrAnHxAAL8MkHRLfeElgzgQxV3Qb+Q== 2024-05-08T00:29:14.315 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-05-08T00:29:14.315 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBQxzpmjj6GKRAADvOOopgZRQ5ojYjsyZVMyQ== 2024-05-08T00:29:14.315 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBTxzpmdBFUKBAAgBkXOxrKePlfkhwCVCSnTw== 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBWxzpmRZQuJxAArHL8pbgT5zS6ahi8M64Chw== 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBNxzpm5gjdIRAAiWfGdw3B0wkeq4jJL4We0Q== 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror] 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBNxzpm60BUJRAAWtACldCHQ+onW+LcL1wqzw== 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.0] 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBNxzpm9+6NKBAAgjrbzwI+/CTaDGUWnTvgPQ== 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.1] 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBNxzpmPZUaLhAA7I6ympig+7zwtvWZYdohbA== 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.2] 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBNxzpmYa+qMxAAi0OgISNw4RcE5D0uLgI17g== 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.3] 2024-05-08T00:29:14.316 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBNxzpm+rEtORAA+HkOjQA/iOxAeRDRaqUL/w== 2024-05-08T00:29:14.317 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.4] 2024-05-08T00:29:14.317 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBOxzpmK3crAxAATxOVMpr1tydbfmfayAJNTA== 2024-05-08T00:29:14.317 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.5] 2024-05-08T00:29:14.317 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBOxzpmlWPKCBAA3KV2B2PW7lkr4mNbQOAYIw== 2024-05-08T00:29:14.317 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.6] 2024-05-08T00:29:14.317 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBOxzpmnb5WDhAAy44UMj2yPMVsbbm6+O2L7Q== 2024-05-08T00:29:14.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-08T00:29:14.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T00:29:14.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T00:29:14.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T00:29:14.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:14.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:14.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:14.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:14.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:14.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:14.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:14.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:15.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:15.067 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-08T00:29:15.067 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-05-08T00:29:15.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-05-08T00:29:15.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-05-08T00:29:15.215 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-05-08T00:29:15.272 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-05-08T00:29:15.272 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-05-08T00:29:15.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-05-08T00:29:15.314 INFO:tasks.ceph.cluster1.mon.a:Started 2024-05-08T00:29:15.314 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-05-08T00:29:15.315 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-05-08T00:29:15.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-05-08T00:29:15.317 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-05-08T00:29:15.317 INFO:tasks.ceph:Setting crush tunables to default 2024-05-08T00:29:15.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster cluster1 osd crush tunables default 2024-05-08T00:29:15.498 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-05-08T00:29:15.509 INFO:tasks.ceph:check_enable_crimson: False 2024-05-08T00:29:15.509 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-05-08T00:29:15.509 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:15.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-05-08T00:29:15.535 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:15.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-05-08T00:29:15.602 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:15.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-05-08T00:29:15.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster cluster1 osd new 66871eb8-7afb-4542-9b36-201bfb221deb 0 2024-05-08T00:29:15.707 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:15.708+0000 7fedfd83a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-08T00:29:15.873 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:15.874+0000 7fedfd83a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-08T00:29:15.892 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-05-08T00:29:15.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster cluster1 osd new d13ef93c-d6a7-46a7-8190-3c39b06d43d5 1 2024-05-08T00:29:15.931 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:15.932+0000 7fedfd83a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-08T00:29:16.091 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-05-08T00:29:16.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster cluster1 osd new 35fef95e-5bb0-4022-8a3a-4b4d9ee78eed 2 2024-05-08T00:29:16.208 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:16.209+0000 7fedfd83a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-08T00:29:16.258 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:16.259+0000 7fedfd83a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-08T00:29:16.328 INFO:teuthology.orchestra.run.smithi064.stdout:2 2024-05-08T00:29:16.341 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-05-08T00:29:16.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-05-08T00:29:16.343 INFO:tasks.ceph.cluster1.osd.0:Started 2024-05-08T00:29:16.343 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-05-08T00:29:16.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-05-08T00:29:16.345 INFO:tasks.ceph.cluster1.osd.1:Started 2024-05-08T00:29:16.345 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-05-08T00:29:16.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-05-08T00:29:16.347 INFO:tasks.ceph.cluster1.osd.2:Started 2024-05-08T00:29:16.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-05-08T00:29:16.531 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:16.531 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"fsid":"4f2229d1-1d9b-4660-ad3b-ce8a6d783878","created":"2024-05-08T00:29:15.407366+0000","modified":"2024-05-08T00:29:16.330023+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T00:29:16.330023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"66871eb8-7afb-4542-9b36-201bfb221deb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d13ef93c-d6a7-46a7-8190-3c39b06d43d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35fef95e-5bb0-4022-8a3a-4b4d9ee78eed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:29:16.543 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-05-08T00:29:16.543 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-08T00:29:16.845 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-05-08T00:29:16.950 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:29:16.948+0000 7f2ef8c94640 -1 Falling back to public interface 2024-05-08T00:29:16.963 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:29:16.965+0000 7fbb1b218640 -1 Falling back to public interface 2024-05-08T00:29:16.976 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:29:16.977+0000 7f532838e640 -1 Falling back to public interface 2024-05-08T00:29:17.019 INFO:teuthology.misc.health.smithi064.stdout: 2024-05-08T00:29:17.019 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":5,"fsid":"4f2229d1-1d9b-4660-ad3b-ce8a6d783878","created":"2024-05-08T00:29:15.407366+0000","modified":"2024-05-08T00:29:16.330023+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T00:29:16.330023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"66871eb8-7afb-4542-9b36-201bfb221deb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d13ef93c-d6a7-46a7-8190-3c39b06d43d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35fef95e-5bb0-4022-8a3a-4b4d9ee78eed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:29:17.033 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-05-08T00:29:17.082 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:17.083+0000 7fedfd83a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-08T00:29:17.174 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:17.175+0000 7fedfd83a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-08T00:29:17.218 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:17.219+0000 7fedfd83a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-08T00:29:17.393 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:17.394+0000 7fedfd83a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-08T00:29:17.451 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:17.452+0000 7fedfd83a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-08T00:29:17.506 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:17.507+0000 7fedfd83a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-08T00:29:17.624 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:17.625+0000 7fedfd83a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-08T00:29:18.026 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:18.027+0000 7fedfd83a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-08T00:29:18.299 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:18.300+0000 7fedfd83a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-08T00:29:18.354 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:18.355+0000 7fedfd83a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-08T00:29:18.476 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:18.477+0000 7fedfd83a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-08T00:29:18.781 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:18.782+0000 7fedfd83a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-08T00:29:18.831 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:18.832+0000 7fedfd83a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-08T00:29:18.996 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:18.997+0000 7fedfd83a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-08T00:29:19.140 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:19.141+0000 7fedfd83a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-08T00:29:19.192 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:19.193+0000 7fedfd83a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-08T00:29:19.266 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:19.267+0000 7fedfd83a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-08T00:29:19.314 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:19.315+0000 7fedfd83a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-08T00:29:19.846 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:19.846+0000 7fedfd83a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-08T00:29:19.967 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:19.967+0000 7fedfd83a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-08T00:29:20.398 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:29:20.398+0000 7f2ef8c94640 -1 osd.1 0 log_to_monitors true 2024-05-08T00:29:20.453 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:29:20.453+0000 7fbb1b218640 -1 osd.2 0 log_to_monitors true 2024-05-08T00:29:20.477 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:29:20.477+0000 7f532838e640 -1 osd.0 0 log_to_monitors true 2024-05-08T00:29:21.028 INFO:tasks.ceph.cluster1.mgr.x.smithi064.stderr:2024-05-08T00:29:21.028+0000 7fed9e49b640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-08T00:29:23.246 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:29:23.246+0000 7f2ef4c2a640 -1 osd.1 0 waiting for initial osdmap 2024-05-08T00:29:23.257 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:29:23.257+0000 7f2eeca4a640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T00:29:23.295 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:29:23.295+0000 7fbb16be7640 -1 osd.2 0 waiting for initial osdmap 2024-05-08T00:29:23.304 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:29:23.305+0000 7fbb0ea07640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T00:29:23.336 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-05-08T00:29:23.343 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:29:23.343+0000 7f5324324640 -1 osd.0 0 waiting for initial osdmap 2024-05-08T00:29:23.361 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:29:23.361+0000 7f531c144640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T00:29:23.625 INFO:teuthology.misc.health.smithi064.stdout: 2024-05-08T00:29:23.625 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":7,"fsid":"4f2229d1-1d9b-4660-ad3b-ce8a6d783878","created":"2024-05-08T00:29:15.407366+0000","modified":"2024-05-08T00:29:22.211892+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T00:29:16.330023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"66871eb8-7afb-4542-9b36-201bfb221deb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d13ef93c-d6a7-46a7-8190-3c39b06d43d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"35fef95e-5bb0-4022-8a3a-4b4d9ee78eed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:29:23.636 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-05-08T00:29:29.940 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-05-08T00:29:30.244 INFO:teuthology.misc.health.smithi064.stdout: 2024-05-08T00:29:30.244 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":12,"fsid":"4f2229d1-1d9b-4660-ad3b-ce8a6d783878","created":"2024-05-08T00:29:15.407366+0000","modified":"2024-05-08T00:29:28.261952+0000","last_up_change":"2024-05-08T00:29:24.251757+0000","last_in_change":"2024-05-08T00:29:16.330023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T00:29:26.220314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"66871eb8-7afb-4542-9b36-201bfb221deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6817","nonce":1089770871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6819","nonce":1089770871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6823","nonce":1089770871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6821","nonce":1089770871}]},"public_addr":"172.21.15.64:6817/1089770871","cluster_addr":"172.21.15.64:6819/1089770871","heartbeat_back_addr":"172.21.15.64:6823/1089770871","heartbeat_front_addr":"172.21.15.64:6821/1089770871","state":["exists","up"]},{"osd":1,"uuid":"d13ef93c-d6a7-46a7-8190-3c39b06d43d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6801","nonce":3957903691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6803","nonce":3957903691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6807","nonce":3957903691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6805","nonce":3957903691}]},"public_addr":"172.21.15.64:6801/3957903691","cluster_addr":"172.21.15.64:6803/3957903691","heartbeat_back_addr":"172.21.15.64:6807/3957903691","heartbeat_front_addr":"172.21.15.64:6805/3957903691","state":["exists","up"]},{"osd":2,"uuid":"35fef95e-5bb0-4022-8a3a-4b4d9ee78eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6809","nonce":467041737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6811","nonce":467041737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6815","nonce":467041737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6813","nonce":467041737}]},"public_addr":"172.21.15.64:6809/467041737","cluster_addr":"172.21.15.64:6811/467041737","heartbeat_back_addr":"172.21.15.64:6815/467041737","heartbeat_front_addr":"172.21.15.64:6813/467041737","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.522965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.442692+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.461612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:29:30.256 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-05-08T00:29:30.256 INFO:tasks.ceph:Creating RBD pool 2024-05-08T00:29:30.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-05-08T00:29:31.278 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'rbd' created 2024-05-08T00:29:31.296 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-08T00:29:32.284 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-08T00:29:32.304 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-05-08T00:29:32.305 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-05-08T00:29:32.305 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-05-08T00:29:32.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-05-08T00:29:32.306 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-08T00:29:32.665 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:32.677 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":421509149},{"type":"v1","addr":"172.21.15.64:6825","nonce":421509149}]},"active_addr":"172.21.15.64:6825/421509149","active_change":"2024-05-08T00:29:19.970658+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1193118370}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":163051158}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3108728538}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1839795962}]}]} 2024-05-08T00:29:32.679 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-05-08T00:29:32.679 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-05-08T00:29:32.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-05-08T00:29:32.978 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:32.978 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":14,"fsid":"4f2229d1-1d9b-4660-ad3b-ce8a6d783878","created":"2024-05-08T00:29:15.407366+0000","modified":"2024-05-08T00:29:32.280018+0000","last_up_change":"2024-05-08T00:29:24.251757+0000","last_in_change":"2024-05-08T00:29:16.330023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T00:29:26.220314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T00:29:30.582717+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"66871eb8-7afb-4542-9b36-201bfb221deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6817","nonce":1089770871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6819","nonce":1089770871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6823","nonce":1089770871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6821","nonce":1089770871}]},"public_addr":"172.21.15.64:6817/1089770871","cluster_addr":"172.21.15.64:6819/1089770871","heartbeat_back_addr":"172.21.15.64:6823/1089770871","heartbeat_front_addr":"172.21.15.64:6821/1089770871","state":["exists","up"]},{"osd":1,"uuid":"d13ef93c-d6a7-46a7-8190-3c39b06d43d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6801","nonce":3957903691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6803","nonce":3957903691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6807","nonce":3957903691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6805","nonce":3957903691}]},"public_addr":"172.21.15.64:6801/3957903691","cluster_addr":"172.21.15.64:6803/3957903691","heartbeat_back_addr":"172.21.15.64:6807/3957903691","heartbeat_front_addr":"172.21.15.64:6805/3957903691","state":["exists","up"]},{"osd":2,"uuid":"35fef95e-5bb0-4022-8a3a-4b4d9ee78eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6809","nonce":467041737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6811","nonce":467041737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6815","nonce":467041737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6813","nonce":467041737}]},"public_addr":"172.21.15.64:6809/467041737","cluster_addr":"172.21.15.64:6811/467041737","heartbeat_back_addr":"172.21.15.64:6815/467041737","heartbeat_front_addr":"172.21.15.64:6813/467041737","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.522965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.442692+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.461612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:29:32.990 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-05-08T00:29:32.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-05-08T00:29:33.288 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:33.288 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":14,"fsid":"4f2229d1-1d9b-4660-ad3b-ce8a6d783878","created":"2024-05-08T00:29:15.407366+0000","modified":"2024-05-08T00:29:32.280018+0000","last_up_change":"2024-05-08T00:29:24.251757+0000","last_in_change":"2024-05-08T00:29:16.330023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T00:29:26.220314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T00:29:30.582717+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"66871eb8-7afb-4542-9b36-201bfb221deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6817","nonce":1089770871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6819","nonce":1089770871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6823","nonce":1089770871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6821","nonce":1089770871}]},"public_addr":"172.21.15.64:6817/1089770871","cluster_addr":"172.21.15.64:6819/1089770871","heartbeat_back_addr":"172.21.15.64:6823/1089770871","heartbeat_front_addr":"172.21.15.64:6821/1089770871","state":["exists","up"]},{"osd":1,"uuid":"d13ef93c-d6a7-46a7-8190-3c39b06d43d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6801","nonce":3957903691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6803","nonce":3957903691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6807","nonce":3957903691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6805","nonce":3957903691}]},"public_addr":"172.21.15.64:6801/3957903691","cluster_addr":"172.21.15.64:6803/3957903691","heartbeat_back_addr":"172.21.15.64:6807/3957903691","heartbeat_front_addr":"172.21.15.64:6805/3957903691","state":["exists","up"]},{"osd":2,"uuid":"35fef95e-5bb0-4022-8a3a-4b4d9ee78eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6809","nonce":467041737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6811","nonce":467041737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6815","nonce":467041737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6813","nonce":467041737}]},"public_addr":"172.21.15.64:6809/467041737","cluster_addr":"172.21.15.64:6811/467041737","heartbeat_back_addr":"172.21.15.64:6815/467041737","heartbeat_front_addr":"172.21.15.64:6813/467041737","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.522965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.442692+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.461612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:29:33.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-05-08T00:29:33.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-05-08T00:29:33.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-05-08T00:29:33.447 INFO:teuthology.orchestra.run.smithi064.stdout:34359738373 2024-05-08T00:29:33.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-05-08T00:29:33.456 INFO:teuthology.orchestra.run.smithi064.stdout:34359738373 2024-05-08T00:29:33.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-05-08T00:29:33.457 INFO:teuthology.orchestra.run.smithi064.stdout:34359738372 2024-05-08T00:29:33.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-05-08T00:29:33.753 INFO:teuthology.orchestra.run.smithi064.stdout:34359738372 2024-05-08T00:29:33.766 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2024-05-08T00:29:33.805 INFO:teuthology.orchestra.run.smithi064.stdout:34359738371 2024-05-08T00:29:33.811 INFO:teuthology.orchestra.run.smithi064.stdout:34359738372 2024-05-08T00:29:33.818 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2024-05-08T00:29:33.824 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-05-08T00:29:34.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-05-08T00:29:34.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-05-08T00:29:34.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-05-08T00:29:35.094 INFO:teuthology.orchestra.run.smithi064.stdout:34359738373 2024-05-08T00:29:35.107 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2024-05-08T00:29:35.107 DEBUG:teuthology.parallel:result is None 2024-05-08T00:29:35.168 INFO:teuthology.orchestra.run.smithi064.stdout:34359738372 2024-05-08T00:29:35.181 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0 2024-05-08T00:29:35.181 DEBUG:teuthology.parallel:result is None 2024-05-08T00:29:35.207 INFO:teuthology.orchestra.run.smithi064.stdout:34359738373 2024-05-08T00:29:35.219 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-05-08T00:29:35.220 DEBUG:teuthology.parallel:result is None 2024-05-08T00:29:35.220 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-05-08T00:29:35.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-08T00:29:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:35.557 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-05-08T00:29:35.569 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-05-08T00:29:33.976123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81552,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033968,"statfs":{"total":289910292480,"available":289826783232,"internally_reserved":0,"allocated":499712,"data_stored":1285482,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.693478"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293450+0000","last_change":"2024-05-08T00:29:32.298717+0000","last_active":"2024-05-08T00:29:33.293450+0000","last_peered":"2024-05-08T00:29:33.293450+0000","last_clean":"2024-05-08T00:29:33.293450+0000","last_became_active":"2024-05-08T00:29:32.298235+0000","last_became_peered":"2024-05-08T00:29:32.298235+0000","last_unstale":"2024-05-08T00:29:33.293450+0000","last_undegraded":"2024-05-08T00:29:33.293450+0000","last_fullsized":"2024-05-08T00:29:33.293450+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:29:00.040409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293390+0000","last_change":"2024-05-08T00:29:32.297238+0000","last_active":"2024-05-08T00:29:33.293390+0000","last_peered":"2024-05-08T00:29:33.293390+0000","last_clean":"2024-05-08T00:29:33.293390+0000","last_became_active":"2024-05-08T00:29:32.296922+0000","last_became_peered":"2024-05-08T00:29:32.296922+0000","last_unstale":"2024-05-08T00:29:33.293390+0000","last_undegraded":"2024-05-08T00:29:33.293390+0000","last_fullsized":"2024-05-08T00:29:33.293390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:35:49.661682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293255+0000","last_change":"2024-05-08T00:29:32.296077+0000","last_active":"2024-05-08T00:29:33.293255+0000","last_peered":"2024-05-08T00:29:33.293255+0000","last_clean":"2024-05-08T00:29:33.293255+0000","last_became_active":"2024-05-08T00:29:32.295815+0000","last_became_peered":"2024-05-08T00:29:32.295815+0000","last_unstale":"2024-05-08T00:29:33.293255+0000","last_undegraded":"2024-05-08T00:29:33.293255+0000","last_fullsized":"2024-05-08T00:29:33.293255+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:48.464488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293300+0000","last_change":"2024-05-08T00:29:32.298850+0000","last_active":"2024-05-08T00:29:33.293300+0000","last_peered":"2024-05-08T00:29:33.293300+0000","last_clean":"2024-05-08T00:29:33.293300+0000","last_became_active":"2024-05-08T00:29:32.298232+0000","last_became_peered":"2024-05-08T00:29:32.298232+0000","last_unstale":"2024-05-08T00:29:33.293300+0000","last_undegraded":"2024-05-08T00:29:33.293300+0000","last_fullsized":"2024-05-08T00:29:33.293300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:37:54.763429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:29:32.299652+0000","last_change":"2024-05-08T00:29:32.297972+0000","last_active":"2024-05-08T00:29:32.299652+0000","last_peered":"2024-05-08T00:29:32.299652+0000","last_clean":"2024-05-08T00:29:32.299652+0000","last_became_active":"2024-05-08T00:29:32.297691+0000","last_became_peered":"2024-05-08T00:29:32.297691+0000","last_unstale":"2024-05-08T00:29:32.299652+0000","last_undegraded":"2024-05-08T00:29:32.299652+0000","last_fullsized":"2024-05-08T00:29:32.299652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:33:38.395490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:29:32.296840+0000","last_change":"2024-05-08T00:29:32.295667+0000","last_active":"2024-05-08T00:29:32.296840+0000","last_peered":"2024-05-08T00:29:32.296840+0000","last_clean":"2024-05-08T00:29:32.296840+0000","last_became_active":"2024-05-08T00:29:32.295343+0000","last_became_peered":"2024-05-08T00:29:32.295343+0000","last_unstale":"2024-05-08T00:29:32.296840+0000","last_undegraded":"2024-05-08T00:29:32.296840+0000","last_fullsized":"2024-05-08T00:29:32.296840+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:30:19.352516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:29:32.298537+0000","last_change":"2024-05-08T00:29:32.297596+0000","last_active":"2024-05-08T00:29:32.298537+0000","last_peered":"2024-05-08T00:29:32.298537+0000","last_clean":"2024-05-08T00:29:32.298537+0000","last_became_active":"2024-05-08T00:29:32.297247+0000","last_became_peered":"2024-05-08T00:29:32.297247+0000","last_unstale":"2024-05-08T00:29:32.298537+0000","last_undegraded":"2024-05-08T00:29:32.298537+0000","last_fullsized":"2024-05-08T00:29:32.298537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:42:18.521881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293244+0000","last_change":"2024-05-08T00:29:32.295731+0000","last_active":"2024-05-08T00:29:33.293244+0000","last_peered":"2024-05-08T00:29:33.293244+0000","last_clean":"2024-05-08T00:29:33.293244+0000","last_became_active":"2024-05-08T00:29:32.295380+0000","last_became_peered":"2024-05-08T00:29:32.295380+0000","last_unstale":"2024-05-08T00:29:33.293244+0000","last_undegraded":"2024-05-08T00:29:33.293244+0000","last_fullsized":"2024-05-08T00:29:33.293244+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:19:18.847185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293188+0000","last_change":"2024-05-08T00:29:27.459396+0000","last_active":"2024-05-08T00:29:33.293188+0000","last_peered":"2024-05-08T00:29:33.293188+0000","last_clean":"2024-05-08T00:29:33.293188+0000","last_became_active":"2024-05-08T00:29:27.458282+0000","last_became_peered":"2024-05-08T00:29:27.458282+0000","last_unstale":"2024-05-08T00:29:33.293188+0000","last_undegraded":"2024-05-08T00:29:33.293188+0000","last_fullsized":"2024-05-08T00:29:33.293188+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_clean_scrub_stamp":"2024-05-08T00:29:26.256116+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:00:43.917924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":627602,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":624124,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T00:29:35.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-08T00:29:35.864 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-05-08T00:29:35.876 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-05-08T00:29:33.976123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81552,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033968,"statfs":{"total":289910292480,"available":289826783232,"internally_reserved":0,"allocated":499712,"data_stored":1285482,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.693478"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293450+0000","last_change":"2024-05-08T00:29:32.298717+0000","last_active":"2024-05-08T00:29:33.293450+0000","last_peered":"2024-05-08T00:29:33.293450+0000","last_clean":"2024-05-08T00:29:33.293450+0000","last_became_active":"2024-05-08T00:29:32.298235+0000","last_became_peered":"2024-05-08T00:29:32.298235+0000","last_unstale":"2024-05-08T00:29:33.293450+0000","last_undegraded":"2024-05-08T00:29:33.293450+0000","last_fullsized":"2024-05-08T00:29:33.293450+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:29:00.040409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293390+0000","last_change":"2024-05-08T00:29:32.297238+0000","last_active":"2024-05-08T00:29:33.293390+0000","last_peered":"2024-05-08T00:29:33.293390+0000","last_clean":"2024-05-08T00:29:33.293390+0000","last_became_active":"2024-05-08T00:29:32.296922+0000","last_became_peered":"2024-05-08T00:29:32.296922+0000","last_unstale":"2024-05-08T00:29:33.293390+0000","last_undegraded":"2024-05-08T00:29:33.293390+0000","last_fullsized":"2024-05-08T00:29:33.293390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:35:49.661682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293255+0000","last_change":"2024-05-08T00:29:32.296077+0000","last_active":"2024-05-08T00:29:33.293255+0000","last_peered":"2024-05-08T00:29:33.293255+0000","last_clean":"2024-05-08T00:29:33.293255+0000","last_became_active":"2024-05-08T00:29:32.295815+0000","last_became_peered":"2024-05-08T00:29:32.295815+0000","last_unstale":"2024-05-08T00:29:33.293255+0000","last_undegraded":"2024-05-08T00:29:33.293255+0000","last_fullsized":"2024-05-08T00:29:33.293255+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:48.464488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293300+0000","last_change":"2024-05-08T00:29:32.298850+0000","last_active":"2024-05-08T00:29:33.293300+0000","last_peered":"2024-05-08T00:29:33.293300+0000","last_clean":"2024-05-08T00:29:33.293300+0000","last_became_active":"2024-05-08T00:29:32.298232+0000","last_became_peered":"2024-05-08T00:29:32.298232+0000","last_unstale":"2024-05-08T00:29:33.293300+0000","last_undegraded":"2024-05-08T00:29:33.293300+0000","last_fullsized":"2024-05-08T00:29:33.293300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:37:54.763429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:29:32.299652+0000","last_change":"2024-05-08T00:29:32.297972+0000","last_active":"2024-05-08T00:29:32.299652+0000","last_peered":"2024-05-08T00:29:32.299652+0000","last_clean":"2024-05-08T00:29:32.299652+0000","last_became_active":"2024-05-08T00:29:32.297691+0000","last_became_peered":"2024-05-08T00:29:32.297691+0000","last_unstale":"2024-05-08T00:29:32.299652+0000","last_undegraded":"2024-05-08T00:29:32.299652+0000","last_fullsized":"2024-05-08T00:29:32.299652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:33:38.395490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:29:32.296840+0000","last_change":"2024-05-08T00:29:32.295667+0000","last_active":"2024-05-08T00:29:32.296840+0000","last_peered":"2024-05-08T00:29:32.296840+0000","last_clean":"2024-05-08T00:29:32.296840+0000","last_became_active":"2024-05-08T00:29:32.295343+0000","last_became_peered":"2024-05-08T00:29:32.295343+0000","last_unstale":"2024-05-08T00:29:32.296840+0000","last_undegraded":"2024-05-08T00:29:32.296840+0000","last_fullsized":"2024-05-08T00:29:32.296840+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:30:19.352516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:29:32.298537+0000","last_change":"2024-05-08T00:29:32.297596+0000","last_active":"2024-05-08T00:29:32.298537+0000","last_peered":"2024-05-08T00:29:32.298537+0000","last_clean":"2024-05-08T00:29:32.298537+0000","last_became_active":"2024-05-08T00:29:32.297247+0000","last_became_peered":"2024-05-08T00:29:32.297247+0000","last_unstale":"2024-05-08T00:29:32.298537+0000","last_undegraded":"2024-05-08T00:29:32.298537+0000","last_fullsized":"2024-05-08T00:29:32.298537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:42:18.521881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293244+0000","last_change":"2024-05-08T00:29:32.295731+0000","last_active":"2024-05-08T00:29:33.293244+0000","last_peered":"2024-05-08T00:29:33.293244+0000","last_clean":"2024-05-08T00:29:33.293244+0000","last_became_active":"2024-05-08T00:29:32.295380+0000","last_became_peered":"2024-05-08T00:29:32.295380+0000","last_unstale":"2024-05-08T00:29:33.293244+0000","last_undegraded":"2024-05-08T00:29:33.293244+0000","last_fullsized":"2024-05-08T00:29:33.293244+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:19:18.847185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-08T00:29:33.293188+0000","last_change":"2024-05-08T00:29:27.459396+0000","last_active":"2024-05-08T00:29:33.293188+0000","last_peered":"2024-05-08T00:29:33.293188+0000","last_clean":"2024-05-08T00:29:33.293188+0000","last_became_active":"2024-05-08T00:29:27.458282+0000","last_became_peered":"2024-05-08T00:29:27.458282+0000","last_unstale":"2024-05-08T00:29:33.293188+0000","last_undegraded":"2024-05-08T00:29:33.293188+0000","last_fullsized":"2024-05-08T00:29:33.293188+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_clean_scrub_stamp":"2024-05-08T00:29:26.256116+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:00:43.917924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":627602,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":624124,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T00:29:35.877 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-05-08T00:29:35.877 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-05-08T00:29:35.877 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-05-08T00:29:35.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-05-08T00:29:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:36.213 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-08T00:29:36.225 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-05-08T00:29:36.225 INFO:teuthology.run_tasks:Running task ceph... 2024-05-08T00:29:36.668 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-05-08T00:29:36.668 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-08T00:29:36.668 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630', 'branch': 'wip-dis-testing-squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7696630', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-squid', 'sha1': '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5306, 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220', 'sleep_before_teardown': 0, 'subset': '21/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNse/AFoy1yDcGyY+MS/1v1sY6rGKOdpmwkIA7spCDsdeG/eq/LWdPvBzKpmx6d2QajiJpoDqcDNFeS5zRkvOOI=', 'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJY4Atyn5pq0iMFwyGZY2SznIWkQuGBB2prrTIQN9ZryfhoD6Ci81KzAnIOQdzmtaHC2/SgzepfCshe0z9Vzcbc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e0c1691aade07c0b78ccff9531f65d3c80e3e220'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-07_23:02:08', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-08T00:29:36.669 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-05-08T00:29:36.672 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-05-08T00:29:36.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-05-08T00:29:36.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-05-08T00:29:36.760 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:36.760 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-05-08T00:29:36.818 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-08T00:29:36.818 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-05-08T00:29:36.875 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-08T00:29:36.875 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T00:29:36.875 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-08T00:29:36.875 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T00:29:36.875 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T00:29:36.876 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-05-08 00:28:25.285911957 +0000 2024-05-08T00:29:36.876 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-05-08 00:25:51.225924926 +0000 2024-05-08T00:29:36.876 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-05-08 00:25:51.225924926 +0000 2024-05-08T00:29:36.876 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-05-08 00:25:51.225924926 +0000 2024-05-08T00:29:36.876 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-08T00:29:36.942 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-05-08T00:29:36.942 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-05-08T00:29:36.942 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000217337 s, 2.4 MB/s 2024-05-08T00:29:36.943 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-08T00:29:37.002 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-05-08T00:29:37.060 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-08T00:29:37.060 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T00:29:37.060 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-08T00:29:37.060 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T00:29:37.060 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T00:29:37.060 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-05-08 00:28:25.286911941 +0000 2024-05-08T00:29:37.060 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-05-08 00:25:51.591918925 +0000 2024-05-08T00:29:37.060 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-05-08 00:25:51.591918925 +0000 2024-05-08T00:29:37.060 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-05-08 00:25:51.591918925 +0000 2024-05-08T00:29:37.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-08T00:29:37.125 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-05-08T00:29:37.126 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-05-08T00:29:37.126 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000200976 s, 2.5 MB/s 2024-05-08T00:29:37.127 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-08T00:29:37.187 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-05-08T00:29:37.243 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-08T00:29:37.243 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T00:29:37.243 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-05-08T00:29:37.243 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T00:29:37.243 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T00:29:37.243 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-05-08 00:28:25.286911941 +0000 2024-05-08T00:29:37.243 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-05-08 00:25:51.917913580 +0000 2024-05-08T00:29:37.244 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-05-08 00:25:51.917913580 +0000 2024-05-08T00:29:37.244 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-05-08 00:25:51.917913580 +0000 2024-05-08T00:29:37.244 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-08T00:29:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-05-08T00:29:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-05-08T00:29:37.308 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000199441 s, 2.6 MB/s 2024-05-08T00:29:37.309 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-08T00:29:37.367 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-05-08T00:29:37.422 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-08T00:29:37.423 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T00:29:37.423 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-08T00:29:37.423 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T00:29:37.423 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-05-08T00:29:37.423 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-05-08 00:28:25.286911941 +0000 2024-05-08T00:29:37.423 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-05-08 00:25:52.253908071 +0000 2024-05-08T00:29:37.423 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-05-08 00:25:52.253908071 +0000 2024-05-08T00:29:37.423 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-05-08 00:25:52.253908071 +0000 2024-05-08T00:29:37.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-08T00:29:37.486 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-05-08T00:29:37.487 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-05-08T00:29:37.487 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000330085 s, 1.6 MB/s 2024-05-08T00:29:37.488 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-08T00:29:37.544 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-08T00:29:37.544 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-08T00:29:37.544 INFO:tasks.ceph:Generating config... 2024-05-08T00:29:37.545 INFO:tasks.ceph:[client] debug rbd = 20 2024-05-08T00:29:37.545 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-05-08T00:29:37.545 INFO:tasks.ceph:[client] log to stderr = False 2024-05-08T00:29:37.545 INFO:tasks.ceph:[client] rbd default features = 125 2024-05-08T00:29:37.545 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:37.545 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:37.545 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:37.546 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:37.546 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-08T00:29:37.546 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-05-08T00:29:37.546 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-08T00:29:37.547 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-08T00:29:37.547 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-08T00:29:37.547 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-08T00:29:37.547 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-08T00:29:37.547 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-08T00:29:37.547 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-08T00:29:37.548 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-08T00:29:37.548 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-08T00:29:37.548 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-08T00:29:37.548 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-08T00:29:37.548 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-05-08T00:29:37.548 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-05-08T00:29:37.623 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster2.keyring 2024-05-08T00:29:37.625 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-05-08T00:29:37.709 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-05-08T00:29:37.774 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.116')] 2024-05-08T00:29:37.774 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.116', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-05-08T00:29:37.775 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:37.775 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-08T00:29:37.829 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.116 --print /home/ubuntu/cephtest/cluster2.monmap 2024-05-08T00:29:37.908 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-05-08T00:29:37.908 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: generated fsid 33ffd8b9-c6fe-4232-9b60-cc9dbec54a30 2024-05-08T00:29:37.908 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = quincy 2024-05-08T00:29:37.909 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 0 2024-05-08T00:29:37.909 INFO:teuthology.orchestra.run.smithi116.stdout:fsid 33ffd8b9-c6fe-4232-9b60-cc9dbec54a30 2024-05-08T00:29:37.909 INFO:teuthology.orchestra.run.smithi116.stdout:last_changed 2024-05-08T00:29:37.910543+0000 2024-05-08T00:29:37.909 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-05-08T00:29:37.910543+0000 2024-05-08T00:29:37.909 INFO:teuthology.orchestra.run.smithi116.stdout:min_mon_release 17 (quincy) 2024-05-08T00:29:37.909 INFO:teuthology.orchestra.run.smithi116.stdout:election_strategy: 1 2024-05-08T00:29:37.909 INFO:teuthology.orchestra.run.smithi116.stdout:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-05-08T00:29:37.909 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-05-08T00:29:37.910 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-08T00:29:37.965 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 33ffd8b9-c6fe-4232-9b60-cc9dbec54a30... 2024-05-08T00:29:37.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-05-08T00:29:37.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-05-08T00:29:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-05-08T00:29:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-05-08T00:29:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = 172.21.15.116 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on pool no app = False 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 33ffd8b9-c6fe-4232-9b60-cc9dbec54a30 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-05-08T00:29:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-05-08T00:29:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = snappy 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 1/20 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 1/20 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 4/10 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-05-08T00:29:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-05-08T00:29:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T00:29:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T00:29:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: debug rbd = 20 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: debug rbd_mirror = 15 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: log to stderr = False 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: rbd default features = 125 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.0] 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.1] 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.2] 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.030 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.3] 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.4] 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.5] 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout:[client.mirror.6] 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.047 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-05-08T00:29:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-05-08T00:29:38.047 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T00:29:38.048 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T00:29:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = 172.21.15.116 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: mon client directed command retry = 5 2024-05-08T00:29:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on pool no app = False 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: ms inject socket failures = 5000 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 33ffd8b9-c6fe-4232-9b60-cc9dbec54a30 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-05-08T00:29:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-05-08T00:29:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore compression algorithm = snappy 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore compression mode = aggressive 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-05-08T00:29:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-05-08T00:29:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-05-08T00:29:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T00:29:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-05-08T00:29:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: debug rbd = 20 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: debug rbd_mirror = 15 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: log to stderr = False 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: rbd default features = 125 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.0] 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.1] 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.2] 2024-05-08T00:29:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.3] 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.4] 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.5] 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.6] 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-08T00:29:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-08T00:29:38.060 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-05-08T00:29:38.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-05-08T00:29:38.155 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-08T00:29:38.155 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:38.155 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-05-08T00:29:38.171 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:38.171 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-05-08T00:29:38.227 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T00:29:38.227 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:38.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/cluster2.keyring 2024-05-08T00:29:38.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-05-08T00:29:38.271 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:38.271 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-05-08T00:29:38.329 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T00:29:38.329 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:38.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/cluster2.keyring 2024-05-08T00:29:38.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-05-08T00:29:38.364 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:38.364 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-05-08T00:29:38.419 INFO:tasks.ceph:Setting up mon nodes... 2024-05-08T00:29:38.419 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-08T00:29:38.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-05-08T00:29:38.479 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-05-08T00:29:38.482 INFO:tasks.ceph:Setting up mds nodes... 2024-05-08T00:29:38.482 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-08T00:29:38.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-05-08T00:29:38.527 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-05-08T00:29:38.541 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-05-08T00:29:38.582 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-05-08T00:29:38.595 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-05-08T00:29:38.675 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-05-08T00:29:38.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-05-08T00:29:38.768 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-05-08T00:29:38.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-05-08T00:29:38.862 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-05-08T00:29:38.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-05-08T00:29:38.955 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-05-08T00:29:38.968 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-05-08T00:29:39.048 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-05-08T00:29:39.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-05-08T00:29:39.141 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-05-08T00:29:39.154 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-05-08T00:29:39.234 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-05-08T00:29:39.247 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-08T00:29:39.247 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-08T00:29:39.247 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-05-08T00:29:39.311 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-08T00:29:39.311 INFO:tasks.ceph:role: cluster2.osd.0 2024-05-08T00:29:39.311 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T00:29:39.311 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T00:29:39.383 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T00:29:39.889 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-05-08T00:29:39.891 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-05-08T00:29:39.891 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-05-08T00:29:40.010 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-05-08T00:29:40.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-05-08T00:29:40.109 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-08T00:29:40.109 INFO:tasks.ceph:role: cluster2.osd.1 2024-05-08T00:29:40.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T00:29:40.110 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-08T00:29:40.182 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T00:29:40.183 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T00:29:40.183 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T00:29:40.183 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-08T00:29:40.183 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T00:29:40.183 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-05-08T00:29:40.183 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T00:29:40.183 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-08T00:29:40.183 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T00:29:40.183 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T00:29:40.646 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-05-08T00:29:40.647 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-05-08T00:29:40.648 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-05-08T00:29:40.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-05-08T00:29:40.797 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-05-08T00:29:40.862 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-08T00:29:40.863 INFO:tasks.ceph:role: cluster2.osd.2 2024-05-08T00:29:40.863 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T00:29:40.863 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T00:29:40.935 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T00:29:41.415 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-05-08T00:29:41.417 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-05-08T00:29:41.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-05-08T00:29:41.569 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-05-08T00:29:41.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-05-08T00:29:41.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:41.778+0000 7fa530d16640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-05-08T00:29:41.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:41.784+0000 7fa530d16640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-05-08T00:29:41.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:41.784+0000 7fa530d16640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-05-08T00:29:41.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:41.784+0000 7fa530d16640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-05-08T00:29:41.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:41.784+0000 7fa530d16640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-05-08T00:29:41.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:41.791+0000 7fa530d16640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-05-08T00:29:44.638 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-05-08T00:29:44.667 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-05-08T00:29:44.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:44.759+0000 7f680c218640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-05-08T00:29:44.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:44.759+0000 7f680c218640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-05-08T00:29:44.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:44.759+0000 7f680c218640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-05-08T00:29:44.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:44.759+0000 7f680c218640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-05-08T00:29:44.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:44.759+0000 7f680c218640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-05-08T00:29:44.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:44.760+0000 7f680c218640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-05-08T00:29:47.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-05-08T00:29:47.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-05-08T00:29:47.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:47.732+0000 7f371822c640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-05-08T00:29:47.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:47.733+0000 7f371822c640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-05-08T00:29:47.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:47.733+0000 7f371822c640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-05-08T00:29:47.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:47.733+0000 7f371822c640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-05-08T00:29:47.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:47.733+0000 7f371822c640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-05-08T00:29:47.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-05-08T00:29:47.733+0000 7f371822c640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-05-08T00:29:50.588 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-05-08T00:29:50.618 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-08T00:29:50.618 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:50.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-05-08T00:29:50.649 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:50.649 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-05-08T00:29:50.685 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:50.685 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-05-08T00:29:50.753 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:50.754 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-05-08T00:29:50.820 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-05-08T00:29:50.820 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-05-08T00:29:50.840 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:50.840 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-05-08T00:29:50.878 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:50.879 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-05-08T00:29:50.935 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:50.935 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-05-08T00:29:50.990 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:50.991 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-05-08T00:29:51.046 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:51.046 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-05-08T00:29:51.102 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:51.102 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-05-08T00:29:51.157 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:51.157 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-05-08T00:29:51.212 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:51.212 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-05-08T00:29:51.267 INFO:tasks.ceph:Adding keys to all mons... 2024-05-08T00:29:51.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/cluster2.keyring 2024-05-08T00:29:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2024-05-08T00:29:51.330 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmNy65HBAAj0GrBPft093mk4cVNIGO6w== 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQB1xzpmSc2DLhAA4gaEeK5DyUMUPCZssgVivw== 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQB4xzpmZQ1SLRAA6gkVsMvJIazswRryXDtd8g== 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQB7xzpmcve3KxAAwvTwMZz4eFEN1nuGu3z1OA== 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpm1VaNHxAA77D6U4a508BRdBl1ogCOUg== 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror] 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmbF/RIhAAU6U1dJmqID/z+5QZ1bzX3w== 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.0] 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmXEddKBAApqHZDIo1jFWslxlBkQM58A== 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.1] 2024-05-08T00:29:51.331 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmfYvtLRAA0K75Ryf5rji75QOMuhbQvA== 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.2] 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmM5iIMxAAgIRXu9BWEbJ52MjwaQUSBA== 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.3] 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmCGsRORAAfYjM/iKdrUzd4rhPa//AlA== 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.4] 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBzxzpmxqX/AhAAdDKM//u15h/EyL8LjUVgDQ== 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.5] 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBzxzpmXr2MCBAAt9EAZvJVFiKpZxhaOO2Fjg== 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.6] 2024-05-08T00:29:51.332 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBzxzpm+o8XDhAARTJZteFOM0FIsafyh8irBg== 2024-05-08T00:29:51.333 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-08T00:29:51.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T00:29:51.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T00:29:51.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T00:29:51.572 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:51.625 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:51.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:51.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:51.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:51.838 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:51.891 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:51.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:51.997 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T00:29:52.049 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-08T00:29:52.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-05-08T00:29:52.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-05-08T00:29:52.170 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-05-08T00:29:52.195 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-05-08T00:29:52.250 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-05-08T00:29:52.251 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-05-08T00:29:52.251 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-05-08T00:29:52.293 INFO:tasks.ceph.cluster2.mon.a:Started 2024-05-08T00:29:52.293 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-05-08T00:29:52.293 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-05-08T00:29:52.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-05-08T00:29:52.296 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-05-08T00:29:52.296 INFO:tasks.ceph:Setting crush tunables to default 2024-05-08T00:29:52.296 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster cluster2 osd crush tunables default 2024-05-08T00:29:52.467 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-05-08T00:29:52.469 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:52.470+0000 7f278cc14040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-08T00:29:52.478 INFO:tasks.ceph:check_enable_crimson: False 2024-05-08T00:29:52.479 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-05-08T00:29:52.479 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:52.479 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-05-08T00:29:52.503 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:52.504 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-05-08T00:29:52.568 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:29:52.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-05-08T00:29:52.588 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:52.589+0000 7f278cc14040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-08T00:29:52.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster cluster2 osd new 3ad08ad1-fedc-43bd-859b-4eb0b0694ac2 0 2024-05-08T00:29:52.637 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:52.638+0000 7f278cc14040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-08T00:29:52.848 INFO:teuthology.orchestra.run.smithi116.stdout:0 2024-05-08T00:29:52.852 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:52.853+0000 7f278cc14040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-08T00:29:52.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster cluster2 osd new b27c4c48-7292-4368-adc1-884d8e896c11 1 2024-05-08T00:29:52.897 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:52.898+0000 7f278cc14040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-08T00:29:53.035 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-05-08T00:29:53.046 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster cluster2 osd new 98458ca4-a008-4a51-b2e3-14a687146fcd 2 2024-05-08T00:29:53.221 INFO:teuthology.orchestra.run.smithi116.stdout:2 2024-05-08T00:29:53.232 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-05-08T00:29:53.232 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-05-08T00:29:53.235 INFO:tasks.ceph.cluster2.osd.0:Started 2024-05-08T00:29:53.235 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-05-08T00:29:53.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-05-08T00:29:53.237 INFO:tasks.ceph.cluster2.osd.1:Started 2024-05-08T00:29:53.237 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-05-08T00:29:53.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-05-08T00:29:53.239 INFO:tasks.ceph.cluster2.osd.2:Started 2024-05-08T00:29:53.239 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-05-08T00:29:53.402 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:53.404+0000 7f278cc14040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-08T00:29:53.411 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:29:53.411 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"fsid":"33ffd8b9-c6fe-4232-9b60-cc9dbec54a30","created":"2024-05-08T00:29:52.383902+0000","modified":"2024-05-08T00:29:53.223004+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T00:29:53.223004+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3ad08ad1-fedc-43bd-859b-4eb0b0694ac2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b27c4c48-7292-4368-adc1-884d8e896c11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"98458ca4-a008-4a51-b2e3-14a687146fcd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:29:53.422 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-05-08T00:29:53.422 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-08T00:29:53.492 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:53.493+0000 7f278cc14040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-08T00:29:53.536 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:53.537+0000 7f278cc14040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-08T00:29:53.708 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:53.709+0000 7f278cc14040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-08T00:29:53.725 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-05-08T00:29:53.754 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:53.755+0000 7f278cc14040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-08T00:29:53.806 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:53.807+0000 7f278cc14040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-08T00:29:53.831 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:29:53.833+0000 7facfce32640 -1 Falling back to public interface 2024-05-08T00:29:53.833 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:29:53.834+0000 7fa302ff3640 -1 Falling back to public interface 2024-05-08T00:29:53.835 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:29:53.836+0000 7fa01a7a3640 -1 Falling back to public interface 2024-05-08T00:29:53.887 INFO:teuthology.misc.health.smithi116.stdout: 2024-05-08T00:29:53.887 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":5,"fsid":"33ffd8b9-c6fe-4232-9b60-cc9dbec54a30","created":"2024-05-08T00:29:52.383902+0000","modified":"2024-05-08T00:29:53.223004+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T00:29:53.223004+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3ad08ad1-fedc-43bd-859b-4eb0b0694ac2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b27c4c48-7292-4368-adc1-884d8e896c11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"98458ca4-a008-4a51-b2e3-14a687146fcd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:29:53.897 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-05-08T00:29:53.921 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:53.923+0000 7f278cc14040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-08T00:29:54.261 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:54.262+0000 7f278cc14040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-08T00:29:54.483 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:54.484+0000 7f278cc14040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-08T00:29:54.535 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:54.536+0000 7f278cc14040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-08T00:29:54.655 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:54.656+0000 7f278cc14040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-08T00:29:54.937 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:54.938+0000 7f278cc14040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-08T00:29:54.983 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:54.984+0000 7f278cc14040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-08T00:29:55.134 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:55.135+0000 7f278cc14040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-08T00:29:55.274 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:55.275+0000 7f278cc14040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-08T00:29:55.325 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:55.326+0000 7f278cc14040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-08T00:29:55.382 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:55.383+0000 7f278cc14040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-08T00:29:55.427 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:55.428+0000 7f278cc14040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-08T00:29:55.837 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:55.838+0000 7f278cc14040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-08T00:29:55.959 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:55.960+0000 7f278cc14040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-08T00:29:56.967 INFO:tasks.ceph.cluster2.mgr.x.smithi064.stderr:2024-05-08T00:29:56.968+0000 7f272d541640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-08T00:29:57.311 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:29:57.312+0000 7facfce32640 -1 osd.1 0 log_to_monitors true 2024-05-08T00:29:57.311 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:29:57.312+0000 7fa302ff3640 -1 osd.0 0 log_to_monitors true 2024-05-08T00:29:57.342 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:29:57.343+0000 7fa01a7a3640 -1 osd.2 0 log_to_monitors true 2024-05-08T00:29:59.952 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:29:59.953+0000 7facf83b2640 -1 osd.1 0 waiting for initial osdmap 2024-05-08T00:29:59.961 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:29:59.962+0000 7fa2fef89640 -1 osd.0 0 waiting for initial osdmap 2024-05-08T00:29:59.966 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:29:59.967+0000 7facf01d2640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T00:29:59.975 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:29:59.976+0000 7fa2f6da9640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T00:29:59.976 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:29:59.977+0000 7fa016739640 -1 osd.2 0 waiting for initial osdmap 2024-05-08T00:29:59.979 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:29:59.980+0000 7fa00e559640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T00:30:00.200 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-05-08T00:30:00.475 INFO:teuthology.misc.health.smithi116.stdout: 2024-05-08T00:30:00.475 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":7,"fsid":"33ffd8b9-c6fe-4232-9b60-cc9dbec54a30","created":"2024-05-08T00:29:52.383902+0000","modified":"2024-05-08T00:29:58.968563+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T00:29:53.223004+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3ad08ad1-fedc-43bd-859b-4eb0b0694ac2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b27c4c48-7292-4368-adc1-884d8e896c11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"98458ca4-a008-4a51-b2e3-14a687146fcd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:30:00.487 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-05-08T00:30:06.789 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-05-08T00:30:07.077 INFO:teuthology.misc.health.smithi116.stdout: 2024-05-08T00:30:07.077 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":11,"fsid":"33ffd8b9-c6fe-4232-9b60-cc9dbec54a30","created":"2024-05-08T00:29:52.383902+0000","modified":"2024-05-08T00:30:04.003297+0000","last_up_change":"2024-05-08T00:30:00.964069+0000","last_in_change":"2024-05-08T00:29:53.223004+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T00:30:01.996254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3ad08ad1-fedc-43bd-859b-4eb0b0694ac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6809","nonce":2736825466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6811","nonce":2736825466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6815","nonce":2736825466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6813","nonce":2736825466}]},"public_addr":"172.21.15.116:6809/2736825466","cluster_addr":"172.21.15.116:6811/2736825466","heartbeat_back_addr":"172.21.15.116:6815/2736825466","heartbeat_front_addr":"172.21.15.116:6813/2736825466","state":["exists","up"]},{"osd":1,"uuid":"b27c4c48-7292-4368-adc1-884d8e896c11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6801","nonce":1570588765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6803","nonce":1570588765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6807","nonce":1570588765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6805","nonce":1570588765}]},"public_addr":"172.21.15.116:6801/1570588765","cluster_addr":"172.21.15.116:6803/1570588765","heartbeat_back_addr":"172.21.15.116:6807/1570588765","heartbeat_front_addr":"172.21.15.116:6805/1570588765","state":["exists","up"]},{"osd":2,"uuid":"98458ca4-a008-4a51-b2e3-14a687146fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6817","nonce":3557803709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6819","nonce":3557803709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6823","nonce":3557803709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6821","nonce":3557803709}]},"public_addr":"172.21.15.116:6817/3557803709","cluster_addr":"172.21.15.116:6819/3557803709","heartbeat_back_addr":"172.21.15.116:6823/3557803709","heartbeat_front_addr":"172.21.15.116:6821/3557803709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.326779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.342584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.379765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:30:07.088 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-05-08T00:30:07.088 INFO:tasks.ceph:Creating RBD pool 2024-05-08T00:30:07.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-05-08T00:30:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'rbd' created 2024-05-08T00:30:08.029 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-08T00:30:09.012 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-08T00:30:09.030 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-05-08T00:30:09.031 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-05-08T00:30:09.031 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-05-08T00:30:09.031 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-05-08T00:30:09.031 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-08T00:30:09.365 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:30:09.376 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3843694080},{"type":"v1","addr":"172.21.15.64:6827","nonce":3843694080}]},"active_addr":"172.21.15.64:6827/3843694080","active_change":"2024-05-08T00:29:55.962519+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3780073719}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1542287348}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3729663090}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":378283221}]}]} 2024-05-08T00:30:09.378 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-05-08T00:30:09.378 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-05-08T00:30:09.378 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-05-08T00:30:09.664 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:30:09.665 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":13,"fsid":"33ffd8b9-c6fe-4232-9b60-cc9dbec54a30","created":"2024-05-08T00:29:52.383902+0000","modified":"2024-05-08T00:30:09.013668+0000","last_up_change":"2024-05-08T00:30:00.964069+0000","last_in_change":"2024-05-08T00:29:53.223004+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T00:30:01.996254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T00:30:07.391983+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3ad08ad1-fedc-43bd-859b-4eb0b0694ac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6809","nonce":2736825466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6811","nonce":2736825466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6815","nonce":2736825466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6813","nonce":2736825466}]},"public_addr":"172.21.15.116:6809/2736825466","cluster_addr":"172.21.15.116:6811/2736825466","heartbeat_back_addr":"172.21.15.116:6815/2736825466","heartbeat_front_addr":"172.21.15.116:6813/2736825466","state":["exists","up"]},{"osd":1,"uuid":"b27c4c48-7292-4368-adc1-884d8e896c11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6801","nonce":1570588765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6803","nonce":1570588765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6807","nonce":1570588765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6805","nonce":1570588765}]},"public_addr":"172.21.15.116:6801/1570588765","cluster_addr":"172.21.15.116:6803/1570588765","heartbeat_back_addr":"172.21.15.116:6807/1570588765","heartbeat_front_addr":"172.21.15.116:6805/1570588765","state":["exists","up"]},{"osd":2,"uuid":"98458ca4-a008-4a51-b2e3-14a687146fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6817","nonce":3557803709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6819","nonce":3557803709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6823","nonce":3557803709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6821","nonce":3557803709}]},"public_addr":"172.21.15.116:6817/3557803709","cluster_addr":"172.21.15.116:6819/3557803709","heartbeat_back_addr":"172.21.15.116:6823/3557803709","heartbeat_front_addr":"172.21.15.116:6821/3557803709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.326779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.342584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.379765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:30:09.677 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-05-08T00:30:09.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-05-08T00:30:09.961 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:30:09.961 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":13,"fsid":"33ffd8b9-c6fe-4232-9b60-cc9dbec54a30","created":"2024-05-08T00:29:52.383902+0000","modified":"2024-05-08T00:30:09.013668+0000","last_up_change":"2024-05-08T00:30:00.964069+0000","last_in_change":"2024-05-08T00:29:53.223004+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T00:30:01.996254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T00:30:07.391983+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3ad08ad1-fedc-43bd-859b-4eb0b0694ac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6809","nonce":2736825466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6811","nonce":2736825466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6815","nonce":2736825466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6813","nonce":2736825466}]},"public_addr":"172.21.15.116:6809/2736825466","cluster_addr":"172.21.15.116:6811/2736825466","heartbeat_back_addr":"172.21.15.116:6815/2736825466","heartbeat_front_addr":"172.21.15.116:6813/2736825466","state":["exists","up"]},{"osd":1,"uuid":"b27c4c48-7292-4368-adc1-884d8e896c11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6801","nonce":1570588765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6803","nonce":1570588765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6807","nonce":1570588765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6805","nonce":1570588765}]},"public_addr":"172.21.15.116:6801/1570588765","cluster_addr":"172.21.15.116:6803/1570588765","heartbeat_back_addr":"172.21.15.116:6807/1570588765","heartbeat_front_addr":"172.21.15.116:6805/1570588765","state":["exists","up"]},{"osd":2,"uuid":"98458ca4-a008-4a51-b2e3-14a687146fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6817","nonce":3557803709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6819","nonce":3557803709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6823","nonce":3557803709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6821","nonce":3557803709}]},"public_addr":"172.21.15.116:6817/3557803709","cluster_addr":"172.21.15.116:6819/3557803709","heartbeat_back_addr":"172.21.15.116:6823/3557803709","heartbeat_front_addr":"172.21.15.116:6821/3557803709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.326779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.342584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.379765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T00:30:09.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-05-08T00:30:09.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-05-08T00:30:09.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-05-08T00:30:10.115 INFO:teuthology.orchestra.run.smithi116.stdout:34359738373 2024-05-08T00:30:10.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-05-08T00:30:10.117 INFO:teuthology.orchestra.run.smithi116.stdout:34359738373 2024-05-08T00:30:10.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-05-08T00:30:10.118 INFO:teuthology.orchestra.run.smithi116.stdout:34359738372 2024-05-08T00:30:10.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-05-08T00:30:10.466 INFO:teuthology.orchestra.run.smithi116.stdout:34359738372 2024-05-08T00:30:10.476 INFO:teuthology.orchestra.run.smithi116.stdout:34359738371 2024-05-08T00:30:10.482 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-05-08T00:30:10.482 INFO:teuthology.orchestra.run.smithi116.stdout:34359738372 2024-05-08T00:30:10.490 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.2 2024-05-08T00:30:10.495 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-05-08T00:30:11.483 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-05-08T00:30:11.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-05-08T00:30:11.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-05-08T00:30:11.810 INFO:teuthology.orchestra.run.smithi116.stdout:34359738372 2024-05-08T00:30:11.822 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-05-08T00:30:11.832 INFO:teuthology.orchestra.run.smithi116.stdout:34359738372 2024-05-08T00:30:11.843 INFO:teuthology.orchestra.run.smithi116.stdout:34359738371 2024-05-08T00:30:11.845 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-05-08T00:30:11.856 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.2 2024-05-08T00:30:12.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-05-08T00:30:12.846 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-05-08T00:30:12.857 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-05-08T00:30:13.177 INFO:teuthology.orchestra.run.smithi116.stdout:34359738374 2024-05-08T00:30:13.187 INFO:teuthology.orchestra.run.smithi116.stdout:34359738374 2024-05-08T00:30:13.190 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738374 for osd.0 2024-05-08T00:30:13.190 DEBUG:teuthology.parallel:result is None 2024-05-08T00:30:13.200 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738374 for osd.1 2024-05-08T00:30:13.200 DEBUG:teuthology.parallel:result is None 2024-05-08T00:30:13.228 INFO:teuthology.orchestra.run.smithi116.stdout:34359738373 2024-05-08T00:30:13.241 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738373 for osd.2 2024-05-08T00:30:13.241 DEBUG:teuthology.parallel:result is None 2024-05-08T00:30:13.241 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-05-08T00:30:13.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-08T00:30:13.562 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:30:13.563 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-05-08T00:30:13.574 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-05-08T00:30:11.969282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81536,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033984,"statfs":{"total":289910292480,"available":289826799616,"internally_reserved":0,"allocated":475136,"data_stored":1278279,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4784,"internal_metadata":82963792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000637"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023456+0000","last_change":"2024-05-08T00:30:09.022863+0000","last_active":"2024-05-08T00:30:10.023456+0000","last_peered":"2024-05-08T00:30:10.023456+0000","last_clean":"2024-05-08T00:30:10.023456+0000","last_became_active":"2024-05-08T00:30:09.022284+0000","last_became_peered":"2024-05-08T00:30:09.022284+0000","last_unstale":"2024-05-08T00:30:10.023456+0000","last_undegraded":"2024-05-08T00:30:10.023456+0000","last_fullsized":"2024-05-08T00:30:10.023456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:22:13.046711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023310+0000","last_change":"2024-05-08T00:30:09.022565+0000","last_active":"2024-05-08T00:30:10.023310+0000","last_peered":"2024-05-08T00:30:10.023310+0000","last_clean":"2024-05-08T00:30:10.023310+0000","last_became_active":"2024-05-08T00:30:09.022263+0000","last_became_peered":"2024-05-08T00:30:09.022263+0000","last_unstale":"2024-05-08T00:30:10.023310+0000","last_undegraded":"2024-05-08T00:30:10.023310+0000","last_fullsized":"2024-05-08T00:30:10.023310+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:16:17.936696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023337+0000","last_change":"2024-05-08T00:30:09.022872+0000","last_active":"2024-05-08T00:30:10.023337+0000","last_peered":"2024-05-08T00:30:10.023337+0000","last_clean":"2024-05-08T00:30:10.023337+0000","last_became_active":"2024-05-08T00:30:09.022298+0000","last_became_peered":"2024-05-08T00:30:09.022298+0000","last_unstale":"2024-05-08T00:30:10.023337+0000","last_undegraded":"2024-05-08T00:30:10.023337+0000","last_fullsized":"2024-05-08T00:30:10.023337+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:03:16.834391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023283+0000","last_change":"2024-05-08T00:30:09.022579+0000","last_active":"2024-05-08T00:30:10.023283+0000","last_peered":"2024-05-08T00:30:10.023283+0000","last_clean":"2024-05-08T00:30:10.023283+0000","last_became_active":"2024-05-08T00:30:09.022205+0000","last_became_peered":"2024-05-08T00:30:09.022205+0000","last_unstale":"2024-05-08T00:30:10.023283+0000","last_undegraded":"2024-05-08T00:30:10.023283+0000","last_fullsized":"2024-05-08T00:30:10.023283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:14:39.237433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.517300+0000","last_change":"2024-05-08T00:30:09.023296+0000","last_active":"2024-05-08T00:30:10.517300+0000","last_peered":"2024-05-08T00:30:10.517300+0000","last_clean":"2024-05-08T00:30:10.517300+0000","last_became_active":"2024-05-08T00:30:09.023003+0000","last_became_peered":"2024-05-08T00:30:09.023003+0000","last_unstale":"2024-05-08T00:30:10.517300+0000","last_undegraded":"2024-05-08T00:30:10.517300+0000","last_fullsized":"2024-05-08T00:30:10.517300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:21:27.320591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T00:30:09.024576+0000","last_change":"2024-05-08T00:30:09.023325+0000","last_active":"2024-05-08T00:30:09.024576+0000","last_peered":"2024-05-08T00:30:09.024576+0000","last_clean":"2024-05-08T00:30:09.024576+0000","last_became_active":"2024-05-08T00:30:09.022878+0000","last_became_peered":"2024-05-08T00:30:09.022878+0000","last_unstale":"2024-05-08T00:30:09.024576+0000","last_undegraded":"2024-05-08T00:30:09.024576+0000","last_fullsized":"2024-05-08T00:30:09.024576+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:48:53.599899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T00:30:09.024926+0000","last_change":"2024-05-08T00:30:09.024010+0000","last_active":"2024-05-08T00:30:09.024926+0000","last_peered":"2024-05-08T00:30:09.024926+0000","last_clean":"2024-05-08T00:30:09.024926+0000","last_became_active":"2024-05-08T00:30:09.023682+0000","last_became_peered":"2024-05-08T00:30:09.023682+0000","last_unstale":"2024-05-08T00:30:09.024926+0000","last_undegraded":"2024-05-08T00:30:09.024926+0000","last_fullsized":"2024-05-08T00:30:09.024926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:06:15.497356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023192+0000","last_change":"2024-05-08T00:30:09.023043+0000","last_active":"2024-05-08T00:30:10.023192+0000","last_peered":"2024-05-08T00:30:10.023192+0000","last_clean":"2024-05-08T00:30:10.023192+0000","last_became_active":"2024-05-08T00:30:09.022369+0000","last_became_peered":"2024-05-08T00:30:09.022369+0000","last_unstale":"2024-05-08T00:30:10.023192+0000","last_undegraded":"2024-05-08T00:30:10.023192+0000","last_fullsized":"2024-05-08T00:30:10.023192+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:39:44.848225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023085+0000","last_change":"2024-05-08T00:30:03.285026+0000","last_active":"2024-05-08T00:30:10.023085+0000","last_peered":"2024-05-08T00:30:10.023085+0000","last_clean":"2024-05-08T00:30:10.023085+0000","last_became_active":"2024-05-08T00:30:03.284046+0000","last_became_peered":"2024-05-08T00:30:03.284046+0000","last_unstale":"2024-05-08T00:30:10.023085+0000","last_undegraded":"2024-05-08T00:30:10.023085+0000","last_fullsized":"2024-05-08T00:30:10.023085+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_clean_scrub_stamp":"2024-05-08T00:30:01.996661+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:24:42.092364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":625201,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":621723,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T00:30:13.575 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-08T00:30:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:30:13.856 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-05-08T00:30:13.868 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-05-08T00:30:11.969282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81536,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033984,"statfs":{"total":289910292480,"available":289826799616,"internally_reserved":0,"allocated":475136,"data_stored":1278279,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":4784,"internal_metadata":82963792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000637"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023456+0000","last_change":"2024-05-08T00:30:09.022863+0000","last_active":"2024-05-08T00:30:10.023456+0000","last_peered":"2024-05-08T00:30:10.023456+0000","last_clean":"2024-05-08T00:30:10.023456+0000","last_became_active":"2024-05-08T00:30:09.022284+0000","last_became_peered":"2024-05-08T00:30:09.022284+0000","last_unstale":"2024-05-08T00:30:10.023456+0000","last_undegraded":"2024-05-08T00:30:10.023456+0000","last_fullsized":"2024-05-08T00:30:10.023456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:22:13.046711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023310+0000","last_change":"2024-05-08T00:30:09.022565+0000","last_active":"2024-05-08T00:30:10.023310+0000","last_peered":"2024-05-08T00:30:10.023310+0000","last_clean":"2024-05-08T00:30:10.023310+0000","last_became_active":"2024-05-08T00:30:09.022263+0000","last_became_peered":"2024-05-08T00:30:09.022263+0000","last_unstale":"2024-05-08T00:30:10.023310+0000","last_undegraded":"2024-05-08T00:30:10.023310+0000","last_fullsized":"2024-05-08T00:30:10.023310+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:16:17.936696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023337+0000","last_change":"2024-05-08T00:30:09.022872+0000","last_active":"2024-05-08T00:30:10.023337+0000","last_peered":"2024-05-08T00:30:10.023337+0000","last_clean":"2024-05-08T00:30:10.023337+0000","last_became_active":"2024-05-08T00:30:09.022298+0000","last_became_peered":"2024-05-08T00:30:09.022298+0000","last_unstale":"2024-05-08T00:30:10.023337+0000","last_undegraded":"2024-05-08T00:30:10.023337+0000","last_fullsized":"2024-05-08T00:30:10.023337+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:03:16.834391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023283+0000","last_change":"2024-05-08T00:30:09.022579+0000","last_active":"2024-05-08T00:30:10.023283+0000","last_peered":"2024-05-08T00:30:10.023283+0000","last_clean":"2024-05-08T00:30:10.023283+0000","last_became_active":"2024-05-08T00:30:09.022205+0000","last_became_peered":"2024-05-08T00:30:09.022205+0000","last_unstale":"2024-05-08T00:30:10.023283+0000","last_undegraded":"2024-05-08T00:30:10.023283+0000","last_fullsized":"2024-05-08T00:30:10.023283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:14:39.237433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.517300+0000","last_change":"2024-05-08T00:30:09.023296+0000","last_active":"2024-05-08T00:30:10.517300+0000","last_peered":"2024-05-08T00:30:10.517300+0000","last_clean":"2024-05-08T00:30:10.517300+0000","last_became_active":"2024-05-08T00:30:09.023003+0000","last_became_peered":"2024-05-08T00:30:09.023003+0000","last_unstale":"2024-05-08T00:30:10.517300+0000","last_undegraded":"2024-05-08T00:30:10.517300+0000","last_fullsized":"2024-05-08T00:30:10.517300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:21:27.320591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T00:30:09.024576+0000","last_change":"2024-05-08T00:30:09.023325+0000","last_active":"2024-05-08T00:30:09.024576+0000","last_peered":"2024-05-08T00:30:09.024576+0000","last_clean":"2024-05-08T00:30:09.024576+0000","last_became_active":"2024-05-08T00:30:09.022878+0000","last_became_peered":"2024-05-08T00:30:09.022878+0000","last_unstale":"2024-05-08T00:30:09.024576+0000","last_undegraded":"2024-05-08T00:30:09.024576+0000","last_fullsized":"2024-05-08T00:30:09.024576+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:48:53.599899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-08T00:30:09.024926+0000","last_change":"2024-05-08T00:30:09.024010+0000","last_active":"2024-05-08T00:30:09.024926+0000","last_peered":"2024-05-08T00:30:09.024926+0000","last_clean":"2024-05-08T00:30:09.024926+0000","last_became_active":"2024-05-08T00:30:09.023682+0000","last_became_peered":"2024-05-08T00:30:09.023682+0000","last_unstale":"2024-05-08T00:30:09.024926+0000","last_undegraded":"2024-05-08T00:30:09.024926+0000","last_fullsized":"2024-05-08T00:30:09.024926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:06:15.497356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023192+0000","last_change":"2024-05-08T00:30:09.023043+0000","last_active":"2024-05-08T00:30:10.023192+0000","last_peered":"2024-05-08T00:30:10.023192+0000","last_clean":"2024-05-08T00:30:10.023192+0000","last_became_active":"2024-05-08T00:30:09.022369+0000","last_became_peered":"2024-05-08T00:30:09.022369+0000","last_unstale":"2024-05-08T00:30:10.023192+0000","last_undegraded":"2024-05-08T00:30:10.023192+0000","last_fullsized":"2024-05-08T00:30:10.023192+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:39:44.848225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-08T00:30:10.023085+0000","last_change":"2024-05-08T00:30:03.285026+0000","last_active":"2024-05-08T00:30:10.023085+0000","last_peered":"2024-05-08T00:30:10.023085+0000","last_clean":"2024-05-08T00:30:10.023085+0000","last_became_active":"2024-05-08T00:30:03.284046+0000","last_became_peered":"2024-05-08T00:30:03.284046+0000","last_unstale":"2024-05-08T00:30:10.023085+0000","last_undegraded":"2024-05-08T00:30:10.023085+0000","last_fullsized":"2024-05-08T00:30:10.023085+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_clean_scrub_stamp":"2024-05-08T00:30:01.996661+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:24:42.092364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":625201,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":621723,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T00:30:13.869 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-05-08T00:30:13.869 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-05-08T00:30:13.869 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-05-08T00:30:13.869 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-05-08T00:30:14.188 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T00:30:14.188 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-08T00:30:14.201 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-05-08T00:30:14.201 INFO:teuthology.run_tasks:Running task exec... 2024-05-08T00:30:14.216 INFO:teuthology.task.exec:Executing custom commands... 2024-05-08T00:30:14.216 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T00:30:14.217 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:14.582 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.4] 2024-05-08T00:30:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBOxzpmK3crAxAATxOVMpr1tydbfmfayAJNTA== 2024-05-08T00:30:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:14.583 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.4 2024-05-08T00:30:14.595 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:14.959 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.5] 2024-05-08T00:30:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBOxzpmlWPKCBAA3KV2B2PW7lkr4mNbQOAYIw== 2024-05-08T00:30:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:14.960 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.5 2024-05-08T00:30:14.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:15.338 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.6] 2024-05-08T00:30:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBOxzpmnb5WDhAAy44UMj2yPMVsbbm6+O2L7Q== 2024-05-08T00:30:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:15.339 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.6 2024-05-08T00:30:15.352 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:15.713 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.4] 2024-05-08T00:30:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBzxzpmxqX/AhAAdDKM//u15h/EyL8LjUVgDQ== 2024-05-08T00:30:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.4 2024-05-08T00:30:15.727 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:16.093 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.5] 2024-05-08T00:30:16.093 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBzxzpmXr2MCBAAt9EAZvJVFiKpZxhaOO2Fjg== 2024-05-08T00:30:16.093 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:16.093 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:16.094 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.5 2024-05-08T00:30:16.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:16.471 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.6] 2024-05-08T00:30:16.471 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBzxzpm+o8XDhAARTJZteFOM0FIsafyh8irBg== 2024-05-08T00:30:16.471 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:16.471 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:16.471 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.6 2024-05-08T00:30:16.485 INFO:teuthology.run_tasks:Running task exec... 2024-05-08T00:30:16.494 INFO:teuthology.task.exec:Executing custom commands... 2024-05-08T00:30:16.494 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T00:30:16.494 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:16.862 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror] 2024-05-08T00:30:16.862 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBNxzpm60BUJRAAWtACldCHQ+onW+LcL1wqzw== 2024-05-08T00:30:16.862 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror-peer" 2024-05-08T00:30:16.862 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror 2024-05-08T00:30:16.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:17.239 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.0] 2024-05-08T00:30:17.239 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBNxzpm9+6NKBAAgjrbzwI+/CTaDGUWnTvgPQ== 2024-05-08T00:30:17.240 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:17.240 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:17.240 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.0 2024-05-08T00:30:17.253 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:17.621 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.1] 2024-05-08T00:30:17.621 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBNxzpmPZUaLhAA7I6ympig+7zwtvWZYdohbA== 2024-05-08T00:30:17.621 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:17.621 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.1 2024-05-08T00:30:17.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:17.998 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.2] 2024-05-08T00:30:17.999 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBNxzpmYa+qMxAAi0OgISNw4RcE5D0uLgI17g== 2024-05-08T00:30:17.999 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:17.999 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:17.999 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.2 2024-05-08T00:30:18.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:18.375 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.3] 2024-05-08T00:30:18.375 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBNxzpm+rEtORAA+HkOjQA/iOxAeRDRaqUL/w== 2024-05-08T00:30:18.375 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:18.375 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:18.375 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.3 2024-05-08T00:30:18.388 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:18.748 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror] 2024-05-08T00:30:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmbF/RIhAAU6U1dJmqID/z+5QZ1bzX3w== 2024-05-08T00:30:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror-peer" 2024-05-08T00:30:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror 2024-05-08T00:30:18.762 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:19.121 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.0] 2024-05-08T00:30:19.121 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmXEddKBAApqHZDIo1jFWslxlBkQM58A== 2024-05-08T00:30:19.121 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:19.122 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:19.122 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.0 2024-05-08T00:30:19.135 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:19.499 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.1] 2024-05-08T00:30:19.499 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmfYvtLRAA0K75Ryf5rji75QOMuhbQvA== 2024-05-08T00:30:19.499 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:19.499 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.1 2024-05-08T00:30:19.512 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:19.876 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.2] 2024-05-08T00:30:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmM5iIMxAAgIRXu9BWEbJ52MjwaQUSBA== 2024-05-08T00:30:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.2 2024-05-08T00:30:19.889 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-08T00:30:20.252 INFO:teuthology.orchestra.run.smithi116.stdout:[client.mirror.3] 2024-05-08T00:30:20.252 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQByxzpmCGsRORAAfYjM/iKdrUzd4rhPa//AlA== 2024-05-08T00:30:20.252 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "profile rbd-mirror" 2024-05-08T00:30:20.252 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "profile rbd" 2024-05-08T00:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.mirror.3 2024-05-08T00:30:20.265 INFO:teuthology.run_tasks:Running task workunit... 2024-05-08T00:30:20.281 INFO:tasks.workunit:Pulling workunits from ref 9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2 2024-05-08T00:30:20.282 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-08T00:30:20.282 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-05-08T00:30:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T00:30:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-05-08T00:30:20.298 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-05-08T00:30:20.352 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-05-08T00:30:20.353 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-05-08T00:30:20.408 INFO:tasks.workunit:timeout=3h 2024-05-08T00:30:20.408 INFO:tasks.workunit:cleanup=True 2024-05-08T00:30:20.408 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2 2024-05-08T00:30:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-05-08T00:31:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Note: switching to '9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2'. 2024-05-08T00:31:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: 2024-05-08T00:31:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-08T00:31:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-08T00:31:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-05-08T00:31:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: 2024-05-08T00:31:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-08T00:31:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: git switch -c 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Or undo this operation with: 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: git switch - 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: 2024-05-08T00:31:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:HEAD is now at 9aa7e81b5f5 Merge branch 'wip-65624-squid' of https://github.com/idryomov/ceph into HEAD 2024-05-08T00:31:06.151 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-05-08T00:31:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-08T00:31:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-05-08T00:31:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-08T00:31:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-08T00:31:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-08T00:31:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-05-08T00:31:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-05-08T00:31:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-08T00:31:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-05-08T00:31:07.111 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-05-08T00:31:07.111 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-05-08T00:31:07.127 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-05-08T00:31:07.128 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-05-08T00:31:07.128 DEBUG:teuthology.orchestra.run.smithi116:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9aa7e81b5f50a61b09e3bd562e7d676fd72d84f2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-05-08T00:31:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ IMAGE_COUNT=50 2024-05-08T00:31:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ export LOCKDEP=0 2024-05-08T00:31:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ LOCKDEP=0 2024-05-08T00:31:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-05-08T00:31:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-05-08T00:31:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ type xmlstarlet 2024-05-08T00:31:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ XMLSTARLET=xmlstarlet 2024-05-08T00:31:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ RBD_MIRROR_INSTANCES=2 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CLUSTER1=cluster1 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CLUSTER2=cluster2 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ PEER_CLUSTER_SUFFIX= 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ POOL=mirror 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ PARENT_POOL=mirror_parent 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ NS1=ns1 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ NS2=ns2 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ TEMPDIR= 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ID=mirror 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ RBD_MIRROR_MODE=journal 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ MIRROR_POOL_MODE=pool 2024-05-08T00:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' journal = snapshot ']' 2024-05-08T00:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-05-08T00:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS='--id mirror' 2024-05-08T00:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ LAST_MIRROR_INSTANCE=1 2024-05-08T00:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-05-08T00:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-05-08T00:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-05-08T00:31:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_BIN=. 2024-05-08T00:31:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_SRC=. 2024-05-08T00:31:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' -e CMakeCache.txt ']' 2024-05-08T00:31:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 -gt 0 ']' 2024-05-08T00:31:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ setup 2024-05-08T00:31:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local c 2024-05-08T00:31:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-05-08T00:31:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ setup_tempdir 2024-05-08T00:31:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ mktemp -d 2024-05-08T00:31:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ TEMPDIR=/tmp/tmp.W1VfRby4SU 2024-05-08T00:31:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -z 1 ']' 2024-05-08T00:31:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ setup_pools cluster1 cluster2 2024-05-08T00:31:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local remote_cluster=cluster2 2024-05-08T00:31:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mon_map_file 2024-05-08T00:31:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mon_addr 2024-05-08T00:31:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local admin_key_file 2024-05-08T00:31:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local uuid 2024-05-08T00:31:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-05-08T00:31:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:pool 'mirror' already exists 2024-05-08T00:31:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-05-08T00:31:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:pool 'mirror_parent' already exists 2024-05-08T00:31:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 pool init mirror 2024-05-08T00:31:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-05-08T00:31:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2024-05-08T00:31:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-05-08T00:31:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-05-08T00:31:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-05-08T00:31:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2024-05-08T00:31:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-05-08T00:31:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -z ']' 2024-05-08T00:31:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -z ']' 2024-05-08T00:31:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ peer_add cluster1 mirror cluster2 2024-05-08T00:31:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local client_cluster=cluster2 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local remote_cluster=cluster2 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local uuid_var_name 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local error_code 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local peer_uuid 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 16 32 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-05-08T00:31:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ peer_uuid=f3790d15-0bce-4a25-bdc6-68fa77e05f61 2024-05-08T00:31:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T00:31:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 17 ']' 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 -eq 0 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror_parent 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local client_cluster=cluster2 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local remote_cluster=cluster2 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local uuid_var_name 2024-05-08T00:31:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local error_code 2024-05-08T00:31:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local peer_uuid 2024-05-08T00:31:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 16 32 2024-05-08T00:31:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:31:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-05-08T00:31:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ peer_uuid=54763dab-3046-43b0-b143-658e701ead2c 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 17 ']' 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 -eq 0 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ setup_pools cluster2 cluster1 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local remote_cluster=cluster1 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mon_map_file 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mon_addr 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local admin_key_file 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local uuid 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-05-08T00:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:pool 'mirror' already exists 2024-05-08T00:31:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-05-08T00:31:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:pool 'mirror_parent' already exists 2024-05-08T00:31:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 pool init mirror 2024-05-08T00:31:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-05-08T00:31:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2024-05-08T00:31:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-05-08T00:31:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-05-08T00:31:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-05-08T00:31:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2024-05-08T00:31:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-05-08T00:31:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -z ']' 2024-05-08T00:31:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -z ']' 2024-05-08T00:31:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ peer_add cluster2 mirror cluster1 2024-05-08T00:31:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local client_cluster=cluster1 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local remote_cluster=cluster1 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local uuid_var_name 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local error_code 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local peer_uuid 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 16 32 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:31:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-05-08T00:31:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ peer_uuid=8673a4e2-9bbc-4d35-ab71-206d5a78bab8 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 17 ']' 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 -eq 0 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror_parent 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local client_cluster=cluster1 2024-05-08T00:31:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local remote_cluster=cluster1 2024-05-08T00:31:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local uuid_var_name 2024-05-08T00:31:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local error_code 2024-05-08T00:31:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local peer_uuid 2024-05-08T00:31:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 16 32 2024-05-08T00:31:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:31:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-05-08T00:31:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ peer_uuid=48ba1062-01fa-4b49-ad5c-afb6982982ca 2024-05-08T00:31:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T00:31:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:31:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 17 ']' 2024-05-08T00:31:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 -eq 0 2024-05-08T00:31:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:31:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n octopus ']' 2024-05-08T00:31:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster1 osd set-require-min-compat-client octopus 2024-05-08T00:31:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:set require_min_compat_client to octopus 2024-05-08T00:31:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T00:31:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster2 osd set-require-min-compat-client octopus 2024-05-08T00:31:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:set require_min_compat_client to octopus 2024-05-08T00:31:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ start_mirrors cluster1 2024-05-08T00:31:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:31:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:31:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ start_mirror cluster1:0 2024-05-08T00:31:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:31:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:31:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:31:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-05-08T00:31:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:31:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ start_mirror cluster1:1 2024-05-08T00:31:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:31:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:31:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:31:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:31:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:31:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:31:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:31:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-05-08T00:31:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ start_mirrors cluster2 2024-05-08T00:31:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:31:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ start_mirror cluster2:0 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2:0 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster2:0 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster2 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster2:0 ']' 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster2 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster2 2024-05-08T00:31:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:31:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:31:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:31:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ start_mirror cluster2:1 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2:1 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster2:1 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster2 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster2:1 ']' 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster2 2024-05-08T00:31:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster2 2024-05-08T00:31:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:31:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:31:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:31:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-05-08T00:31:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-05-08T00:31:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ tee -a /tmp/tmp.W1VfRby4SU/rbd-mirror.test.log 2024-05-08T00:31:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ date '+%F %T' 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo 2024-05-08 00:31:26 TEST: add image and test replay after client crashes 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:2024-05-08 00:31:26 TEST: add image and test replay after client crashes 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=test 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 512M 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror test 512M 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:31:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/test 2024-05-08T00:31:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:31:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:31:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:31:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:31:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:31:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:31:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:31:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:31:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:31:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:31:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:31:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:31:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:31:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:31:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:31:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:31:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:31:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:31:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:31:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:31:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:31:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:31:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:31:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:31:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:31:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:31:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:31:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:31:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-08T00:31:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:31:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:31:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:31:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:31:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:31:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:31:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:31:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:31:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:31:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:31:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:31:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:31:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:31:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:31:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-08T00:31:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-08T00:31:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:31:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:31:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:31:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 1 2024-05-08T00:31:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:31:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:31:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:31:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:31:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:31:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:31:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:31:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:31:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:31:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:31:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:31:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:31:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:31:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:31:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-08T00:31:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:31:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:31:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:31:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-08T00:31:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:31:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:31:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:31:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:31:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:31:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:31:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:31:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:31:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:31:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:31:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-08T00:31:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-08T00:31:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:31:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:31:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:31:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 2 2024-05-08T00:31:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:31:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:31:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:31:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:31:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:31:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:31:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:31:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:31:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:31:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:31:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:31:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:31:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:31:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:31:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:31:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:31:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:31:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:31:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:31:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:31:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name= 2024-05-08T00:31:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 10 2024-05-08T00:31:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:31:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:31:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:31:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:31:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=6 2024-05-08T00:31:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:31:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 26600448 bytes 2024-05-08T00:31:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 87571456 bytes 2024-05-08T00:31:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 101981184 bytes 2024-05-08T00:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 103685120 bytes 2024-05-08T00:31:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 180051968 bytes 2024-05-08T00:31:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 226856960 bytes 2024-05-08T00:31:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:31:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:31:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:31:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:31:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:31:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:31:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:31:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:31:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:31:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:31:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:31:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:31:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:31:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:31:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-05-08T00:31:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:31:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:31:30 2024-05-08T00:31:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:31:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:31:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:31:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:31:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:31:36' 2024-05-08T00:31:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:31:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:31:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:31:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-05-08T00:31:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:31:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:31:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:31:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:31:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:31:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:31:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:31:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:31:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:31:30' 2024-05-08T00:31:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:31:30' 2024-05-08T00:31:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:31:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:31:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:31:30' 2024-05-08T00:31:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:31:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:31:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:31:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42594266mirror.0smithi1162024-05-08 00:31:30cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:31:36' 2024-05-08T00:31:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:31:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:31:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:31:30' 2024-05-08T00:31:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:31:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:31:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:31:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:31:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:31:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:31:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:31:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap1 2024-05-08T00:31:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap1 2024-05-08T00:31:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:31:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:31:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:31:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap1 2024-05-08T00:31:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:32:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:32:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:32:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:32:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:32:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:32:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:32:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:32:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:32:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:32:08.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:32:08.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:32:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:32:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:32:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:32:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:32:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:32:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:32:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:32:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=14, tag_tid=2, entry_tid=13462], [object_number=15, tag_tid=2, entry_tid=13459]]], state=connected] 2024-05-08T00:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=14, tag_tid=2, entry_tid=13462], [object_number=15, tag_tid=2, entry_tid=13459]]], state=connected] 2024-05-08T00:32:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-05-08T00:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:32:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:32:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=14, tag_tid=2, entry_tid=13462], [object_number=15, tag_tid=2, entry_tid=13459]]], state=connected] 2024-05-08T00:32:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=14, tag_tid=2, entry_tid=13462], [object_number=15, tag_tid=2, entry_tid=13459]]], state=connected] 2024-05-08T00:32:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-05-08T00:32:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=3, entry_tid=1' 2024-05-08T00:32:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-05-08T00:32:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap1 2024-05-08T00:32:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:32:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:32:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:32:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-05-08T00:32:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:32:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:32:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:32:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:32:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:32:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:32:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:32:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:32:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap1 2024-05-08T00:32:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap1 2024-05-08T00:32:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:32:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:32:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:32:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:32:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:32:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap1-clean 2024-05-08T00:32:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-05-08T00:32:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:32:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap1-clean 2024-05-08T00:32:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:32:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:32:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:32:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:32:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:32:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=6 2024-05-08T00:32:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:32:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:32:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 35043328 bytes 2024-05-08T00:32:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 103003136 bytes 2024-05-08T00:32:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 208110592 bytes 2024-05-08T00:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:32:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:32:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:32:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:32:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:32:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:32:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:32:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:32:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:32:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:32:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:32:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":6238047.3,"entries_behind_primary":5345,"entries_per_second":393.95,"non_primary_position":{"entry_tid":6079,"object_number":7,"tag_tid":2},"primary_position":{"entry_tid":11424,"object_number":12,"tag_tid":2},"seconds_until_synced":13} 2024-05-08T00:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:32:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:31:56 2024-05-08T00:32:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:32:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:32:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:32:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:32:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:32:16' 2024-05-08T00:32:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:32:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:32:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:32:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":6238047.3,"entries_behind_primary":5345,"entries_per_second":393.95,"non_primary_position":{"entry_tid":6079,"object_number":7,"tag_tid":2},"primary_position":{"entry_tid":11424,"object_number":12,"tag_tid":2},"seconds_until_synced":13} 2024-05-08T00:32:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:32:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:32:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:32:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:32:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:32:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:32:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:32:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:32:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:31:56' 2024-05-08T00:32:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:31:56' 2024-05-08T00:32:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:32:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:31:56' 2024-05-08T00:32:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:32:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:32:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:32:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":6238047.3,"entries_behind_primary":5345,"entries_per_second":393.95,"non_primary_position":{"entry_tid":6079,"object_number":7,"tag_tid":2},"primary_position":{"entry_tid":11424,"object_number":12,"tag_tid":2},"seconds_until_synced":13}42594266mirror.0smithi1162024-05-08 00:31:56cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:32:16' 2024-05-08T00:32:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:32:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:32:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:31:56' 2024-05-08T00:32:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:32:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:32:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:32:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:32:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap2 2024-05-08T00:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap2 2024-05-08T00:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap2 2024-05-08T00:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:32:37.692 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:32:37.693+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:32:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:32:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:32:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:32:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:32:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:32:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:32:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:32:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:32:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:32:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:32:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:32:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:32:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:32:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:32:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:32:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:32:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:32:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:32:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:32:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:32:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:32:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:32:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 7 2024-05-08T00:32:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=29, tag_tid=5, entry_tid=14225], [object_number=28, tag_tid=5, entry_tid=14224], [object_number=31, tag_tid=5, entry_tid=14223], [object_number=30, tag_tid=5, entry_tid=14222]]], state=connected] 2024-05-08T00:32:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 7 2024-05-08T00:32:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=29, tag_tid=5, entry_tid=14225], [object_number=28, tag_tid=5, entry_tid=14224], [object_number=31, tag_tid=5, entry_tid=14223], [object_number=30, tag_tid=5, entry_tid=14222]]], state=connected] 2024-05-08T00:32:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=29, tag_tid=5, entry_tid=14225' 2024-05-08T00:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=29, tag_tid=5, entry_tid=14225' 2024-05-08T00:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=29, tag_tid=5, entry_tid=14225' '!=' '' 2024-05-08T00:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=29, tag_tid=5, entry_tid=14225' = '' 2024-05-08T00:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=29, tag_tid=5, entry_tid=14225' 2024-05-08T00:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=6 2024-05-08T00:32:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=29, tag_tid=5, entry_tid=14225' 2024-05-08T00:32:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=5 2024-05-08T00:32:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:32:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=29, tag_tid=5, entry_tid=14225' 2024-05-08T00:32:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14225 2024-05-08T00:32:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 6 = 5 -a 1 -le 14225 2024-05-08T00:32:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:32:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:32:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 7 2024-05-08T00:32:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=31, tag_tid=5, entry_tid=14395], [object_number=30, tag_tid=5, entry_tid=14394], [object_number=29, tag_tid=5, entry_tid=14393], [object_number=28, tag_tid=5, entry_tid=14392]]], state=connected] 2024-05-08T00:32:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 7 2024-05-08T00:32:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=31, tag_tid=5, entry_tid=14395], [object_number=30, tag_tid=5, entry_tid=14394], [object_number=29, tag_tid=5, entry_tid=14393], [object_number=28, tag_tid=5, entry_tid=14392]]], state=connected] 2024-05-08T00:32:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=31, tag_tid=5, entry_tid=14395' 2024-05-08T00:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=31, tag_tid=5, entry_tid=14395' 2024-05-08T00:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=31, tag_tid=5, entry_tid=14395' '!=' 'object_number=29, tag_tid=5, entry_tid=14225' 2024-05-08T00:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=31, tag_tid=5, entry_tid=14395' = 'object_number=29, tag_tid=5, entry_tid=14225' 2024-05-08T00:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=31, tag_tid=5, entry_tid=14395' 2024-05-08T00:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=6 2024-05-08T00:32:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=31, tag_tid=5, entry_tid=14395' 2024-05-08T00:32:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=5 2024-05-08T00:32:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:32:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=31, tag_tid=5, entry_tid=14395' 2024-05-08T00:32:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14395 2024-05-08T00:32:50.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 6 = 5 -a 1 -le 14395 2024-05-08T00:32:50.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:32:50.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:50.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:32:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 7 2024-05-08T00:32:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=28, tag_tid=5, entry_tid=14560], [object_number=31, tag_tid=5, entry_tid=14559], [object_number=30, tag_tid=5, entry_tid=14558], [object_number=29, tag_tid=5, entry_tid=14557]]], state=connected] 2024-05-08T00:32:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 7 2024-05-08T00:32:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=28, tag_tid=5, entry_tid=14560], [object_number=31, tag_tid=5, entry_tid=14559], [object_number=30, tag_tid=5, entry_tid=14558], [object_number=29, tag_tid=5, entry_tid=14557]]], state=connected] 2024-05-08T00:32:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=28, tag_tid=5, entry_tid=14560' 2024-05-08T00:32:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=28, tag_tid=5, entry_tid=14560' 2024-05-08T00:32:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=28, tag_tid=5, entry_tid=14560' '!=' 'object_number=31, tag_tid=5, entry_tid=14395' 2024-05-08T00:32:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:32:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=28, tag_tid=5, entry_tid=14560' = 'object_number=31, tag_tid=5, entry_tid=14395' 2024-05-08T00:32:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=28, tag_tid=5, entry_tid=14560' 2024-05-08T00:32:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=6 2024-05-08T00:32:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=28, tag_tid=5, entry_tid=14560' 2024-05-08T00:32:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=5 2024-05-08T00:32:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:32:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=28, tag_tid=5, entry_tid=14560' 2024-05-08T00:32:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14560 2024-05-08T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 6 = 5 -a 1 -le 14560 2024-05-08T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:50.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:50.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:50.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 7 2024-05-08T00:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=30, tag_tid=5, entry_tid=14742], [object_number=29, tag_tid=5, entry_tid=14741], [object_number=28, tag_tid=5, entry_tid=14740], [object_number=31, tag_tid=5, entry_tid=14739]]], state=connected] 2024-05-08T00:32:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 7 2024-05-08T00:32:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=30, tag_tid=5, entry_tid=14742], [object_number=29, tag_tid=5, entry_tid=14741], [object_number=28, tag_tid=5, entry_tid=14740], [object_number=31, tag_tid=5, entry_tid=14739]]], state=connected] 2024-05-08T00:32:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=30, tag_tid=5, entry_tid=14742' 2024-05-08T00:32:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=30, tag_tid=5, entry_tid=14742' 2024-05-08T00:32:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=30, tag_tid=5, entry_tid=14742' '!=' 'object_number=28, tag_tid=5, entry_tid=14560' 2024-05-08T00:32:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:32:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=30, tag_tid=5, entry_tid=14742' = 'object_number=28, tag_tid=5, entry_tid=14560' 2024-05-08T00:32:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=30, tag_tid=5, entry_tid=14742' 2024-05-08T00:32:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=6 2024-05-08T00:32:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=30, tag_tid=5, entry_tid=14742' 2024-05-08T00:32:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=5 2024-05-08T00:32:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:32:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=30, tag_tid=5, entry_tid=14742' 2024-05-08T00:32:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14742 2024-05-08T00:32:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 6 = 5 -a 1 -le 14742 2024-05-08T00:32:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:32:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:32:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=14921], [object_number=32, tag_tid=5, entry_tid=14920], [object_number=35, tag_tid=5, entry_tid=14919], [object_number=34, tag_tid=5, entry_tid=14918]]], state=connected] 2024-05-08T00:32:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=14921], [object_number=32, tag_tid=5, entry_tid=14920], [object_number=35, tag_tid=5, entry_tid=14919], [object_number=34, tag_tid=5, entry_tid=14918]]], state=connected] 2024-05-08T00:32:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=14921' 2024-05-08T00:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=14921' 2024-05-08T00:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=14921' '!=' 'object_number=30, tag_tid=5, entry_tid=14742' 2024-05-08T00:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=14921' = 'object_number=30, tag_tid=5, entry_tid=14742' 2024-05-08T00:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=14921' 2024-05-08T00:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=6 2024-05-08T00:32:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=14921' 2024-05-08T00:32:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=5 2024-05-08T00:32:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:32:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=14921' 2024-05-08T00:32:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14921 2024-05-08T00:32:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 6 = 5 -a 1 -le 14921 2024-05-08T00:32:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:32:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=15084], [object_number=35, tag_tid=5, entry_tid=15083], [object_number=34, tag_tid=5, entry_tid=15082], [object_number=33, tag_tid=5, entry_tid=15081]]], state=connected] 2024-05-08T00:32:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=15084], [object_number=35, tag_tid=5, entry_tid=15083], [object_number=34, tag_tid=5, entry_tid=15082], [object_number=33, tag_tid=5, entry_tid=15081]]], state=connected] 2024-05-08T00:32:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:51.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=15084' 2024-05-08T00:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=15084' 2024-05-08T00:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=15084' '!=' 'object_number=33, tag_tid=5, entry_tid=14921' 2024-05-08T00:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=15084' = 'object_number=33, tag_tid=5, entry_tid=14921' 2024-05-08T00:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=15084' 2024-05-08T00:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=6 2024-05-08T00:32:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=15084' 2024-05-08T00:32:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=5 2024-05-08T00:32:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:32:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=15084' 2024-05-08T00:32:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15084 2024-05-08T00:32:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 6 = 5 -a 1 -le 15084 2024-05-08T00:32:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:32:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:32:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110], [object_number=33, tag_tid=5, entry_tid=15109], [object_number=32, tag_tid=5, entry_tid=15108]]], state=connected] 2024-05-08T00:32:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110], [object_number=33, tag_tid=5, entry_tid=15109], [object_number=32, tag_tid=5, entry_tid=15108]]], state=connected] 2024-05-08T00:32:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=15111' 2024-05-08T00:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=15111' 2024-05-08T00:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=15111' '!=' 'object_number=32, tag_tid=5, entry_tid=15084' 2024-05-08T00:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=15111' = 'object_number=32, tag_tid=5, entry_tid=15084' 2024-05-08T00:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=15111' 2024-05-08T00:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=6 2024-05-08T00:32:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=15111' 2024-05-08T00:32:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:32:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=5 2024-05-08T00:32:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:32:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=15111' 2024-05-08T00:32:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:32:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:32:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15111 2024-05-08T00:32:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 6 = 5 -a 1 -le 15111 2024-05-08T00:32:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:32:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:32:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110], [object_number=33, tag_tid=5, entry_tid=15109], [object_number=32, tag_tid=5, entry_tid=15108]]], state=connected] 2024-05-08T00:32:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110], [object_number=33, tag_tid=5, entry_tid=15109], [object_number=32, tag_tid=5, entry_tid=15108]]], state=connected] 2024-05-08T00:32:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=15111' 2024-05-08T00:32:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=15111' 2024-05-08T00:32:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=15111' '!=' 'object_number=35, tag_tid=5, entry_tid=15111' 2024-05-08T00:32:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:32:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.4 2024-05-08T00:32:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:32:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:32:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:32:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:32:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:32:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:32:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:32:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:32:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:32:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:32:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:32:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:32:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:32:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:32:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:32:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 8 2024-05-08T00:32:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 8 2024-05-08T00:32:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:32:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=35, tag_tid=5, entry_tid=15111], [object_number=34, tag_tid=5, entry_tid=15110]]], state=connected] 2024-05-08T00:32:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=6, entry_tid=1' 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap2 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:32:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-05-08T00:32:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:32:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:32:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:32:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 3 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:32:29 2024 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:32:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n snap1-clean ']' 2024-05-08T00:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap1-clean 2024-05-08T00:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap1-clean 2024-05-08T00:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:32:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:32:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:32:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap2 2024-05-08T00:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap2 2024-05-08T00:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:32:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:32:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:56.064 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:32:56.065+0000 7f530a106640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-05-08T00:32:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:32:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:32:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:32:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap2-clean 2024-05-08T00:32:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-05-08T00:32:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:32:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap2-clean 2024-05-08T00:32:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:32:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:32:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:32:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:32:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:32:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:32:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:32:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:32:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:32:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:32:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 52940800 bytes 2024-05-08T00:33:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 87025664 bytes 2024-05-08T00:33:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 105795584 bytes 2024-05-08T00:33:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 149096448 bytes 2024-05-08T00:33:03.041 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:33:03.042+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:33:03.041 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:33:03.042+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:33:03.041 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:33:03.042+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22000 2024-05-08T00:33:03.041 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:33:03.042+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:33:03.042 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:33:03.043+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:33:03.042 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:33:03.043+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:33:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 196332544 bytes 2024-05-08T00:33:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:33:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:33:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:33:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:33:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:33:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:33:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:33:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:33:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:33:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":6230474.5,"entries_behind_primary":0,"entries_per_second":385.0,"non_primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6}} 2024-05-08T00:33:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:33:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:32:56 2024-05-08T00:33:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:33:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:33:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:33:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:33:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:33:06' 2024-05-08T00:33:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:33:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:33:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:33:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":6230474.5,"entries_behind_primary":0,"entries_per_second":385.0,"non_primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6}} 2024-05-08T00:33:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:33:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:33:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:33:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:33:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:33:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:33:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:33:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:33:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:32:56' 2024-05-08T00:33:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:32:56' 2024-05-08T00:33:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:33:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:32:56' 2024-05-08T00:33:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:33:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:33:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:33:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":6230474.5,"entries_behind_primary":0,"entries_per_second":385.0,"non_primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6}}42594266mirror.0smithi1162024-05-08 00:32:56cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:33:06' 2024-05-08T00:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:33:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:32:56' 2024-05-08T00:33:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:33:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:33:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:33:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:33:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:33:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap3 2024-05-08T00:33:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap3 2024-05-08T00:33:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:33:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap3 2024-05-08T00:33:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:33:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:33:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:33:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:33:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:33:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:33:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:33:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:33:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:33:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:33:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:33:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:33:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:33:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:33:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:33:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:33:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:33:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:33:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:33:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:33:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:33:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:33:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:33:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:33:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:33:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:33:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:33:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:33:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:33:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:33:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:33:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:33:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:33:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:33:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:33:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:33:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:33:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:33:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:33:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:33:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:33:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:33:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:33:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:33:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:33:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:33:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:33:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:33:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:33:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:33:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:33:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:33:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:33:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:33:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:33:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:33:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:33:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:33:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 12 2024-05-08T00:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 12 2024-05-08T00:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=9, entry_tid=1], [object_number=48, tag_tid=9, entry_tid=0], [object_number=51, tag_tid=8, entry_tid=14715], [object_number=50, tag_tid=8, entry_tid=14714]]], state=connected] 2024-05-08T00:33:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=49, tag_tid=9, entry_tid=1], [object_number=48, tag_tid=9, entry_tid=0], [object_number=51, tag_tid=8, entry_tid=14715], [object_number=50, tag_tid=8, entry_tid=14714]]], state=connected] 2024-05-08T00:33:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=49, tag_tid=9, entry_tid=1' 2024-05-08T00:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:33:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:33:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:33:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:33:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:33:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:33:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:33:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:33:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:33:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 12 2024-05-08T00:33:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 12 2024-05-08T00:33:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:33:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=49, tag_tid=9, entry_tid=1], [object_number=48, tag_tid=9, entry_tid=0], [object_number=51, tag_tid=8, entry_tid=14715], [object_number=50, tag_tid=8, entry_tid=14714]]], state=connected] 2024-05-08T00:33:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=49, tag_tid=9, entry_tid=1], [object_number=48, tag_tid=9, entry_tid=0], [object_number=51, tag_tid=8, entry_tid=14715], [object_number=50, tag_tid=8, entry_tid=14714]]], state=connected] 2024-05-08T00:33:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=49, tag_tid=9, entry_tid=1' 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=49, tag_tid=9, entry_tid=1' -a 'object_number=49, tag_tid=9, entry_tid=1' = 'object_number=49, tag_tid=9, entry_tid=1' 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap3 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:33:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:33:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 5 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:33:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:32:54 2024 2024-05-08T00:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:33:30 2024 2024-05-08T00:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n snap2-clean ']' 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap2-clean 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap2-clean 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:33:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:33:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:41.417 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:33:41.417+0000 7f53060fe640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-05-08T00:33:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:33:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap3 2024-05-08T00:33:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap3 2024-05-08T00:33:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:33:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:33:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:33:42.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:33:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:33:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:33:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap3-clean 2024-05-08T00:33:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-05-08T00:33:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:33:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap3-clean 2024-05-08T00:33:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:33:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:33:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:33:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:33:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:33:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=12 2024-05-08T00:33:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:33:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:33:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 50529280 bytes 2024-05-08T00:33:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 97209344 bytes 2024-05-08T00:33:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 182270976 bytes 2024-05-08T00:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 214559744 bytes 2024-05-08T00:33:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 241057792 bytes 2024-05-08T00:33:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 284296192 bytes 2024-05-08T00:33:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:700 IOs, 323726336 bytes 2024-05-08T00:33:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:33:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:33:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:33:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:33:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:33:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:33:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:33:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:33:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:33:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:33:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":4597536.0,"entries_behind_primary":7634,"entries_per_second":286.57,"non_primary_position":{"entry_tid":504,"object_number":48,"tag_tid":11},"primary_position":{"entry_tid":8138,"object_number":58,"tag_tid":11},"seconds_until_synced":26} 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:33:56 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:33:56' 2024-05-08T00:33:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:33:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:33:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:33:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":4597536.0,"entries_behind_primary":7634,"entries_per_second":286.57,"non_primary_position":{"entry_tid":504,"object_number":48,"tag_tid":11},"primary_position":{"entry_tid":8138,"object_number":58,"tag_tid":11},"seconds_until_synced":26} 2024-05-08T00:33:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:33:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:33:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:33:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:33:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:33:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:33:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:33:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:33:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:33:56' 2024-05-08T00:33:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:33:56' 2024-05-08T00:33:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:33:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:33:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:33:56' 2024-05-08T00:33:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:33:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:33:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:33:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":4597536.0,"entries_behind_primary":7634,"entries_per_second":286.57,"non_primary_position":{"entry_tid":504,"object_number":48,"tag_tid":11},"primary_position":{"entry_tid":8138,"object_number":58,"tag_tid":11},"seconds_until_synced":26}42594266mirror.0smithi1162024-05-08 00:33:56cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:33:56' 2024-05-08T00:33:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:33:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:33:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:33:56' 2024-05-08T00:33:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:33:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:33:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:33:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:33:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:33:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:33:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap4 2024-05-08T00:33:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap4 2024-05-08T00:33:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:33:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:33:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:33:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap4 2024-05-08T00:33:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:34:15.616 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:34:15.617+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:34:15.617 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:34:15.617+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:34:19.028 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:34:19.029+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0ba00 2024-05-08T00:34:19.029 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:34:19.030+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:34:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:34:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:34:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:34:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:34:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:34:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:34:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:34:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:34:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:34:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:34:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:34:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:34:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:34:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:34:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:34:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:34:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:34:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:34:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 16 2024-05-08T00:34:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=15904], [object_number=67, tag_tid=11, entry_tid=15903], [object_number=66, tag_tid=11, entry_tid=15902], [object_number=65, tag_tid=11, entry_tid=15901]]], state=connected] 2024-05-08T00:34:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 16 2024-05-08T00:34:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=15904], [object_number=67, tag_tid=11, entry_tid=15903], [object_number=66, tag_tid=11, entry_tid=15902], [object_number=65, tag_tid=11, entry_tid=15901]]], state=connected] 2024-05-08T00:34:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=15904' 2024-05-08T00:34:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=15904' 2024-05-08T00:34:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=15904' '!=' '' 2024-05-08T00:34:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=15904' = '' 2024-05-08T00:34:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=15904' 2024-05-08T00:34:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=15904' 2024-05-08T00:34:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=15904' 2024-05-08T00:34:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15904 2024-05-08T00:34:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 15904 2024-05-08T00:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 16 2024-05-08T00:34:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=16095], [object_number=66, tag_tid=11, entry_tid=16094], [object_number=65, tag_tid=11, entry_tid=16093], [object_number=64, tag_tid=11, entry_tid=16092]]], state=connected] 2024-05-08T00:34:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 16 2024-05-08T00:34:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=16095], [object_number=66, tag_tid=11, entry_tid=16094], [object_number=65, tag_tid=11, entry_tid=16093], [object_number=64, tag_tid=11, entry_tid=16092]]], state=connected] 2024-05-08T00:34:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=16095' 2024-05-08T00:34:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=16095' 2024-05-08T00:34:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=16095' '!=' 'object_number=64, tag_tid=11, entry_tid=15904' 2024-05-08T00:34:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=16095' = 'object_number=64, tag_tid=11, entry_tid=15904' 2024-05-08T00:34:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=16095' 2024-05-08T00:34:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=16095' 2024-05-08T00:34:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=16095' 2024-05-08T00:34:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16095 2024-05-08T00:34:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 16095 2024-05-08T00:34:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 16 2024-05-08T00:34:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=16262], [object_number=65, tag_tid=11, entry_tid=16261], [object_number=64, tag_tid=11, entry_tid=16260], [object_number=67, tag_tid=11, entry_tid=16259]]], state=connected] 2024-05-08T00:34:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 16 2024-05-08T00:34:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=16262], [object_number=65, tag_tid=11, entry_tid=16261], [object_number=64, tag_tid=11, entry_tid=16260], [object_number=67, tag_tid=11, entry_tid=16259]]], state=connected] 2024-05-08T00:34:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=66, tag_tid=11, entry_tid=16262' 2024-05-08T00:34:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=66, tag_tid=11, entry_tid=16262' 2024-05-08T00:34:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=16262' '!=' 'object_number=67, tag_tid=11, entry_tid=16095' 2024-05-08T00:34:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=16262' = 'object_number=67, tag_tid=11, entry_tid=16095' 2024-05-08T00:34:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=66, tag_tid=11, entry_tid=16262' 2024-05-08T00:34:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=16262' 2024-05-08T00:34:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=16262' 2024-05-08T00:34:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16262 2024-05-08T00:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 16262 2024-05-08T00:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=16297], [object_number=68, tag_tid=11, entry_tid=16296], [object_number=71, tag_tid=11, entry_tid=16295], [object_number=70, tag_tid=11, entry_tid=16294]]], state=connected] 2024-05-08T00:34:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=16297], [object_number=68, tag_tid=11, entry_tid=16296], [object_number=71, tag_tid=11, entry_tid=16295], [object_number=70, tag_tid=11, entry_tid=16294]]], state=connected] 2024-05-08T00:34:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=16297' 2024-05-08T00:34:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=16297' 2024-05-08T00:34:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=16297' '!=' 'object_number=66, tag_tid=11, entry_tid=16262' 2024-05-08T00:34:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=16297' = 'object_number=66, tag_tid=11, entry_tid=16262' 2024-05-08T00:34:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=16297' 2024-05-08T00:34:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=16297' 2024-05-08T00:34:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=16297' 2024-05-08T00:34:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16297 2024-05-08T00:34:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 16297 2024-05-08T00:34:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=16453], [object_number=68, tag_tid=11, entry_tid=16452], [object_number=71, tag_tid=11, entry_tid=16451], [object_number=70, tag_tid=11, entry_tid=16450]]], state=connected] 2024-05-08T00:34:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=16453], [object_number=68, tag_tid=11, entry_tid=16452], [object_number=71, tag_tid=11, entry_tid=16451], [object_number=70, tag_tid=11, entry_tid=16450]]], state=connected] 2024-05-08T00:34:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=16453' 2024-05-08T00:34:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=16453' 2024-05-08T00:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=16453' '!=' 'object_number=69, tag_tid=11, entry_tid=16297' 2024-05-08T00:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=16453' = 'object_number=69, tag_tid=11, entry_tid=16297' 2024-05-08T00:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=16453' 2024-05-08T00:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=16453' 2024-05-08T00:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=16453' 2024-05-08T00:34:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16453 2024-05-08T00:34:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 16453 2024-05-08T00:34:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=16603], [object_number=70, tag_tid=11, entry_tid=16602], [object_number=69, tag_tid=11, entry_tid=16601], [object_number=68, tag_tid=11, entry_tid=16600]]], state=connected] 2024-05-08T00:34:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=16603], [object_number=70, tag_tid=11, entry_tid=16602], [object_number=69, tag_tid=11, entry_tid=16601], [object_number=68, tag_tid=11, entry_tid=16600]]], state=connected] 2024-05-08T00:34:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=16603' 2024-05-08T00:34:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=16603' 2024-05-08T00:34:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=16603' '!=' 'object_number=69, tag_tid=11, entry_tid=16453' 2024-05-08T00:34:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=16603' = 'object_number=69, tag_tid=11, entry_tid=16453' 2024-05-08T00:34:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=16603' 2024-05-08T00:34:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=16603' 2024-05-08T00:34:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=16603' 2024-05-08T00:34:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16603 2024-05-08T00:34:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 16603 2024-05-08T00:34:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=16839], [object_number=70, tag_tid=11, entry_tid=16838], [object_number=69, tag_tid=11, entry_tid=16837], [object_number=68, tag_tid=11, entry_tid=16836]]], state=connected] 2024-05-08T00:34:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=16839], [object_number=70, tag_tid=11, entry_tid=16838], [object_number=69, tag_tid=11, entry_tid=16837], [object_number=68, tag_tid=11, entry_tid=16836]]], state=connected] 2024-05-08T00:34:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=16839' 2024-05-08T00:34:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=16839' 2024-05-08T00:34:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=16839' '!=' 'object_number=71, tag_tid=11, entry_tid=16603' 2024-05-08T00:34:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=16839' = 'object_number=71, tag_tid=11, entry_tid=16603' 2024-05-08T00:34:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=16839' 2024-05-08T00:34:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=16839' 2024-05-08T00:34:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=16839' 2024-05-08T00:34:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16839 2024-05-08T00:34:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 16839 2024-05-08T00:34:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=16994], [object_number=69, tag_tid=11, entry_tid=16993], [object_number=68, tag_tid=11, entry_tid=16992], [object_number=71, tag_tid=11, entry_tid=16991]]], state=connected] 2024-05-08T00:34:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=16994], [object_number=69, tag_tid=11, entry_tid=16993], [object_number=68, tag_tid=11, entry_tid=16992], [object_number=71, tag_tid=11, entry_tid=16991]]], state=connected] 2024-05-08T00:34:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=16994' 2024-05-08T00:34:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=16994' 2024-05-08T00:34:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=16994' '!=' 'object_number=71, tag_tid=11, entry_tid=16839' 2024-05-08T00:34:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=16994' = 'object_number=71, tag_tid=11, entry_tid=16839' 2024-05-08T00:34:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=16994' 2024-05-08T00:34:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=16994' 2024-05-08T00:34:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=16994' 2024-05-08T00:34:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16994 2024-05-08T00:34:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 16994 2024-05-08T00:34:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=17178], [object_number=69, tag_tid=11, entry_tid=17177], [object_number=68, tag_tid=11, entry_tid=17176], [object_number=71, tag_tid=11, entry_tid=17175]]], state=connected] 2024-05-08T00:34:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=17178], [object_number=69, tag_tid=11, entry_tid=17177], [object_number=68, tag_tid=11, entry_tid=17176], [object_number=71, tag_tid=11, entry_tid=17175]]], state=connected] 2024-05-08T00:34:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=17178' 2024-05-08T00:34:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=17178' 2024-05-08T00:34:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=17178' '!=' 'object_number=70, tag_tid=11, entry_tid=16994' 2024-05-08T00:34:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=17178' = 'object_number=70, tag_tid=11, entry_tid=16994' 2024-05-08T00:34:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=17178' 2024-05-08T00:34:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=17178' 2024-05-08T00:34:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=17178' 2024-05-08T00:34:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17178 2024-05-08T00:34:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 17178 2024-05-08T00:34:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:33.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=17418], [object_number=69, tag_tid=11, entry_tid=17417], [object_number=68, tag_tid=11, entry_tid=17416], [object_number=71, tag_tid=11, entry_tid=17415]]], state=connected] 2024-05-08T00:34:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=17418], [object_number=69, tag_tid=11, entry_tid=17417], [object_number=68, tag_tid=11, entry_tid=17416], [object_number=71, tag_tid=11, entry_tid=17415]]], state=connected] 2024-05-08T00:34:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=17418' 2024-05-08T00:34:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=17418' 2024-05-08T00:34:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=17418' '!=' 'object_number=70, tag_tid=11, entry_tid=17178' 2024-05-08T00:34:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=17418' = 'object_number=70, tag_tid=11, entry_tid=17178' 2024-05-08T00:34:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=17418' 2024-05-08T00:34:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=17418' 2024-05-08T00:34:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:34.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=17418' 2024-05-08T00:34:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17418 2024-05-08T00:34:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 17418 2024-05-08T00:34:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=17583], [object_number=70, tag_tid=11, entry_tid=17582], [object_number=69, tag_tid=11, entry_tid=17581], [object_number=68, tag_tid=11, entry_tid=17580]]], state=connected] 2024-05-08T00:34:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=17583], [object_number=70, tag_tid=11, entry_tid=17582], [object_number=69, tag_tid=11, entry_tid=17581], [object_number=68, tag_tid=11, entry_tid=17580]]], state=connected] 2024-05-08T00:34:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=17583' 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=17583' 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=17583' '!=' 'object_number=70, tag_tid=11, entry_tid=17418' 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=17583' = 'object_number=70, tag_tid=11, entry_tid=17418' 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=17583' 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=17583' 2024-05-08T00:34:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=17583' 2024-05-08T00:34:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17583 2024-05-08T00:34:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 17583 2024-05-08T00:34:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=17758], [object_number=69, tag_tid=11, entry_tid=17757], [object_number=68, tag_tid=11, entry_tid=17756], [object_number=71, tag_tid=11, entry_tid=17755]]], state=connected] 2024-05-08T00:34:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=17758], [object_number=69, tag_tid=11, entry_tid=17757], [object_number=68, tag_tid=11, entry_tid=17756], [object_number=71, tag_tid=11, entry_tid=17755]]], state=connected] 2024-05-08T00:34:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=17758' 2024-05-08T00:34:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=17758' 2024-05-08T00:34:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=17758' '!=' 'object_number=71, tag_tid=11, entry_tid=17583' 2024-05-08T00:34:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=17758' = 'object_number=71, tag_tid=11, entry_tid=17583' 2024-05-08T00:34:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=17758' 2024-05-08T00:34:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=17758' 2024-05-08T00:34:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=17758' 2024-05-08T00:34:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17758 2024-05-08T00:34:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 17758 2024-05-08T00:34:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=17893], [object_number=68, tag_tid=11, entry_tid=17892], [object_number=71, tag_tid=11, entry_tid=17891], [object_number=70, tag_tid=11, entry_tid=17890]]], state=connected] 2024-05-08T00:34:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=17893], [object_number=68, tag_tid=11, entry_tid=17892], [object_number=71, tag_tid=11, entry_tid=17891], [object_number=70, tag_tid=11, entry_tid=17890]]], state=connected] 2024-05-08T00:34:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=17893' 2024-05-08T00:34:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=17893' 2024-05-08T00:34:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=17893' '!=' 'object_number=70, tag_tid=11, entry_tid=17758' 2024-05-08T00:34:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=17893' = 'object_number=70, tag_tid=11, entry_tid=17758' 2024-05-08T00:34:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=17893' 2024-05-08T00:34:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=17893' 2024-05-08T00:34:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=17893' 2024-05-08T00:34:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17893 2024-05-08T00:34:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 17893 2024-05-08T00:34:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=18053], [object_number=68, tag_tid=11, entry_tid=18052], [object_number=71, tag_tid=11, entry_tid=18051], [object_number=70, tag_tid=11, entry_tid=18050]]], state=connected] 2024-05-08T00:34:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=18053], [object_number=68, tag_tid=11, entry_tid=18052], [object_number=71, tag_tid=11, entry_tid=18051], [object_number=70, tag_tid=11, entry_tid=18050]]], state=connected] 2024-05-08T00:34:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=18053' 2024-05-08T00:34:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=18053' 2024-05-08T00:34:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=18053' '!=' 'object_number=69, tag_tid=11, entry_tid=17893' 2024-05-08T00:34:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=18053' = 'object_number=69, tag_tid=11, entry_tid=17893' 2024-05-08T00:34:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=18053' 2024-05-08T00:34:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=18053' 2024-05-08T00:34:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=18053' 2024-05-08T00:34:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18053 2024-05-08T00:34:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 18053 2024-05-08T00:34:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=18231], [object_number=70, tag_tid=11, entry_tid=18230], [object_number=69, tag_tid=11, entry_tid=18229], [object_number=68, tag_tid=11, entry_tid=18228]]], state=connected] 2024-05-08T00:34:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=71, tag_tid=11, entry_tid=18231], [object_number=70, tag_tid=11, entry_tid=18230], [object_number=69, tag_tid=11, entry_tid=18229], [object_number=68, tag_tid=11, entry_tid=18228]]], state=connected] 2024-05-08T00:34:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=71, tag_tid=11, entry_tid=18231' 2024-05-08T00:34:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=71, tag_tid=11, entry_tid=18231' 2024-05-08T00:34:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=18231' '!=' 'object_number=69, tag_tid=11, entry_tid=18053' 2024-05-08T00:34:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=71, tag_tid=11, entry_tid=18231' = 'object_number=69, tag_tid=11, entry_tid=18053' 2024-05-08T00:34:36.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=71, tag_tid=11, entry_tid=18231' 2024-05-08T00:34:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=18231' 2024-05-08T00:34:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=71, tag_tid=11, entry_tid=18231' 2024-05-08T00:34:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18231 2024-05-08T00:34:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 18231 2024-05-08T00:34:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=18408], [object_number=71, tag_tid=11, entry_tid=18407], [object_number=70, tag_tid=11, entry_tid=18406], [object_number=69, tag_tid=11, entry_tid=18405]]], state=connected] 2024-05-08T00:34:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=18408], [object_number=71, tag_tid=11, entry_tid=18407], [object_number=70, tag_tid=11, entry_tid=18406], [object_number=69, tag_tid=11, entry_tid=18405]]], state=connected] 2024-05-08T00:34:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=68, tag_tid=11, entry_tid=18408' 2024-05-08T00:34:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=68, tag_tid=11, entry_tid=18408' 2024-05-08T00:34:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=18408' '!=' 'object_number=71, tag_tid=11, entry_tid=18231' 2024-05-08T00:34:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=18408' = 'object_number=71, tag_tid=11, entry_tid=18231' 2024-05-08T00:34:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=68, tag_tid=11, entry_tid=18408' 2024-05-08T00:34:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=18408' 2024-05-08T00:34:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=18408' 2024-05-08T00:34:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18408 2024-05-08T00:34:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 18408 2024-05-08T00:34:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=18585], [object_number=68, tag_tid=11, entry_tid=18584], [object_number=71, tag_tid=11, entry_tid=18583], [object_number=70, tag_tid=11, entry_tid=18582]]], state=connected] 2024-05-08T00:34:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=18585], [object_number=68, tag_tid=11, entry_tid=18584], [object_number=71, tag_tid=11, entry_tid=18583], [object_number=70, tag_tid=11, entry_tid=18582]]], state=connected] 2024-05-08T00:34:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=18585' 2024-05-08T00:34:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=18585' 2024-05-08T00:34:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=18585' '!=' 'object_number=68, tag_tid=11, entry_tid=18408' 2024-05-08T00:34:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=18585' = 'object_number=68, tag_tid=11, entry_tid=18408' 2024-05-08T00:34:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=18585' 2024-05-08T00:34:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=18585' 2024-05-08T00:34:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=18585' 2024-05-08T00:34:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18585 2024-05-08T00:34:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 18585 2024-05-08T00:34:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=18769], [object_number=68, tag_tid=11, entry_tid=18768], [object_number=71, tag_tid=11, entry_tid=18767], [object_number=70, tag_tid=11, entry_tid=18766]]], state=connected] 2024-05-08T00:34:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=18769], [object_number=68, tag_tid=11, entry_tid=18768], [object_number=71, tag_tid=11, entry_tid=18767], [object_number=70, tag_tid=11, entry_tid=18766]]], state=connected] 2024-05-08T00:34:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=18769' 2024-05-08T00:34:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=18769' 2024-05-08T00:34:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=18769' '!=' 'object_number=69, tag_tid=11, entry_tid=18585' 2024-05-08T00:34:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=18769' = 'object_number=69, tag_tid=11, entry_tid=18585' 2024-05-08T00:34:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=18769' 2024-05-08T00:34:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=18769' 2024-05-08T00:34:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=18769' 2024-05-08T00:34:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18769 2024-05-08T00:34:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 18769 2024-05-08T00:34:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=18964], [object_number=71, tag_tid=11, entry_tid=18963], [object_number=70, tag_tid=11, entry_tid=18962], [object_number=69, tag_tid=11, entry_tid=18961]]], state=connected] 2024-05-08T00:34:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=18964], [object_number=71, tag_tid=11, entry_tid=18963], [object_number=70, tag_tid=11, entry_tid=18962], [object_number=69, tag_tid=11, entry_tid=18961]]], state=connected] 2024-05-08T00:34:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=68, tag_tid=11, entry_tid=18964' 2024-05-08T00:34:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=68, tag_tid=11, entry_tid=18964' 2024-05-08T00:34:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=18964' '!=' 'object_number=69, tag_tid=11, entry_tid=18769' 2024-05-08T00:34:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=18964' = 'object_number=69, tag_tid=11, entry_tid=18769' 2024-05-08T00:34:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=68, tag_tid=11, entry_tid=18964' 2024-05-08T00:34:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=18964' 2024-05-08T00:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=18964' 2024-05-08T00:34:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18964 2024-05-08T00:34:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 18964 2024-05-08T00:34:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:38.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:38.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=19234], [object_number=69, tag_tid=11, entry_tid=19233], [object_number=68, tag_tid=11, entry_tid=19232], [object_number=71, tag_tid=11, entry_tid=19231]]], state=connected] 2024-05-08T00:34:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=70, tag_tid=11, entry_tid=19234], [object_number=69, tag_tid=11, entry_tid=19233], [object_number=68, tag_tid=11, entry_tid=19232], [object_number=71, tag_tid=11, entry_tid=19231]]], state=connected] 2024-05-08T00:34:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=70, tag_tid=11, entry_tid=19234' 2024-05-08T00:34:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=70, tag_tid=11, entry_tid=19234' 2024-05-08T00:34:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=19234' '!=' 'object_number=68, tag_tid=11, entry_tid=18964' 2024-05-08T00:34:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=70, tag_tid=11, entry_tid=19234' = 'object_number=68, tag_tid=11, entry_tid=18964' 2024-05-08T00:34:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=70, tag_tid=11, entry_tid=19234' 2024-05-08T00:34:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=19234' 2024-05-08T00:34:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=70, tag_tid=11, entry_tid=19234' 2024-05-08T00:34:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=19234 2024-05-08T00:34:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 19234 2024-05-08T00:34:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=19392], [object_number=71, tag_tid=11, entry_tid=19391], [object_number=70, tag_tid=11, entry_tid=19390], [object_number=69, tag_tid=11, entry_tid=19389]]], state=connected] 2024-05-08T00:34:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=68, tag_tid=11, entry_tid=19392], [object_number=71, tag_tid=11, entry_tid=19391], [object_number=70, tag_tid=11, entry_tid=19390], [object_number=69, tag_tid=11, entry_tid=19389]]], state=connected] 2024-05-08T00:34:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=68, tag_tid=11, entry_tid=19392' 2024-05-08T00:34:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=68, tag_tid=11, entry_tid=19392' 2024-05-08T00:34:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=19392' '!=' 'object_number=70, tag_tid=11, entry_tid=19234' 2024-05-08T00:34:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=68, tag_tid=11, entry_tid=19392' = 'object_number=70, tag_tid=11, entry_tid=19234' 2024-05-08T00:34:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=68, tag_tid=11, entry_tid=19392' 2024-05-08T00:34:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=19392' 2024-05-08T00:34:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=68, tag_tid=11, entry_tid=19392' 2024-05-08T00:34:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=19392 2024-05-08T00:34:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 19392 2024-05-08T00:34:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=19549], [object_number=68, tag_tid=11, entry_tid=19548], [object_number=71, tag_tid=11, entry_tid=19547], [object_number=70, tag_tid=11, entry_tid=19546]]], state=connected] 2024-05-08T00:34:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=69, tag_tid=11, entry_tid=19549], [object_number=68, tag_tid=11, entry_tid=19548], [object_number=71, tag_tid=11, entry_tid=19547], [object_number=70, tag_tid=11, entry_tid=19546]]], state=connected] 2024-05-08T00:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=69, tag_tid=11, entry_tid=19549' 2024-05-08T00:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=69, tag_tid=11, entry_tid=19549' 2024-05-08T00:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=19549' '!=' 'object_number=68, tag_tid=11, entry_tid=19392' 2024-05-08T00:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=69, tag_tid=11, entry_tid=19549' = 'object_number=68, tag_tid=11, entry_tid=19392' 2024-05-08T00:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=69, tag_tid=11, entry_tid=19549' 2024-05-08T00:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=19549' 2024-05-08T00:34:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=69, tag_tid=11, entry_tid=19549' 2024-05-08T00:34:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=19549 2024-05-08T00:34:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 19549 2024-05-08T00:34:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=19655], [object_number=74, tag_tid=11, entry_tid=19654], [object_number=69, tag_tid=11, entry_tid=19653], [object_number=68, tag_tid=11, entry_tid=19652]]], state=connected] 2024-05-08T00:34:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 17 2024-05-08T00:34:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=19655], [object_number=74, tag_tid=11, entry_tid=19654], [object_number=69, tag_tid=11, entry_tid=19653], [object_number=68, tag_tid=11, entry_tid=19652]]], state=connected] 2024-05-08T00:34:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=19655' 2024-05-08T00:34:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=19655' 2024-05-08T00:34:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=19655' '!=' 'object_number=69, tag_tid=11, entry_tid=19549' 2024-05-08T00:34:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=19655' = 'object_number=69, tag_tid=11, entry_tid=19549' 2024-05-08T00:34:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=19655' 2024-05-08T00:34:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=19655' 2024-05-08T00:34:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=19655' 2024-05-08T00:34:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=19655 2024-05-08T00:34:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 19655 2024-05-08T00:34:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=19787], [object_number=74, tag_tid=11, entry_tid=19786], [object_number=73, tag_tid=11, entry_tid=19785], [object_number=72, tag_tid=11, entry_tid=19784]]], state=connected] 2024-05-08T00:34:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=19787], [object_number=74, tag_tid=11, entry_tid=19786], [object_number=73, tag_tid=11, entry_tid=19785], [object_number=72, tag_tid=11, entry_tid=19784]]], state=connected] 2024-05-08T00:34:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=19787' 2024-05-08T00:34:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=19787' 2024-05-08T00:34:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=19787' '!=' 'object_number=75, tag_tid=11, entry_tid=19655' 2024-05-08T00:34:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=19787' = 'object_number=75, tag_tid=11, entry_tid=19655' 2024-05-08T00:34:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=19787' 2024-05-08T00:34:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=19787' 2024-05-08T00:34:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=19787' 2024-05-08T00:34:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=19787 2024-05-08T00:34:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 19787 2024-05-08T00:34:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=19965], [object_number=72, tag_tid=11, entry_tid=19964], [object_number=75, tag_tid=11, entry_tid=19963], [object_number=74, tag_tid=11, entry_tid=19962]]], state=connected] 2024-05-08T00:34:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=19965], [object_number=72, tag_tid=11, entry_tid=19964], [object_number=75, tag_tid=11, entry_tid=19963], [object_number=74, tag_tid=11, entry_tid=19962]]], state=connected] 2024-05-08T00:34:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=19965' 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=19965' 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=19965' '!=' 'object_number=75, tag_tid=11, entry_tid=19787' 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=19965' = 'object_number=75, tag_tid=11, entry_tid=19787' 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=19965' 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=19965' 2024-05-08T00:34:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=19965' 2024-05-08T00:34:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=19965 2024-05-08T00:34:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 19965 2024-05-08T00:34:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=20126], [object_number=73, tag_tid=11, entry_tid=20125], [object_number=72, tag_tid=11, entry_tid=20124], [object_number=75, tag_tid=11, entry_tid=20123]]], state=connected] 2024-05-08T00:34:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=20126], [object_number=73, tag_tid=11, entry_tid=20125], [object_number=72, tag_tid=11, entry_tid=20124], [object_number=75, tag_tid=11, entry_tid=20123]]], state=connected] 2024-05-08T00:34:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=20126' 2024-05-08T00:34:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=20126' 2024-05-08T00:34:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=20126' '!=' 'object_number=73, tag_tid=11, entry_tid=19965' 2024-05-08T00:34:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=20126' = 'object_number=73, tag_tid=11, entry_tid=19965' 2024-05-08T00:34:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=20126' 2024-05-08T00:34:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=20126' 2024-05-08T00:34:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=20126' 2024-05-08T00:34:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=20126 2024-05-08T00:34:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 20126 2024-05-08T00:34:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=20293], [object_number=72, tag_tid=11, entry_tid=20292], [object_number=75, tag_tid=11, entry_tid=20291], [object_number=74, tag_tid=11, entry_tid=20290]]], state=connected] 2024-05-08T00:34:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=20293], [object_number=72, tag_tid=11, entry_tid=20292], [object_number=75, tag_tid=11, entry_tid=20291], [object_number=74, tag_tid=11, entry_tid=20290]]], state=connected] 2024-05-08T00:34:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=20293' 2024-05-08T00:34:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=20293' 2024-05-08T00:34:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=20293' '!=' 'object_number=74, tag_tid=11, entry_tid=20126' 2024-05-08T00:34:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=20293' = 'object_number=74, tag_tid=11, entry_tid=20126' 2024-05-08T00:34:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=20293' 2024-05-08T00:34:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=20293' 2024-05-08T00:34:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=20293' 2024-05-08T00:34:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=20293 2024-05-08T00:34:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 20293 2024-05-08T00:34:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=20476], [object_number=75, tag_tid=11, entry_tid=20475], [object_number=74, tag_tid=11, entry_tid=20474], [object_number=73, tag_tid=11, entry_tid=20473]]], state=connected] 2024-05-08T00:34:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=20476], [object_number=75, tag_tid=11, entry_tid=20475], [object_number=74, tag_tid=11, entry_tid=20474], [object_number=73, tag_tid=11, entry_tid=20473]]], state=connected] 2024-05-08T00:34:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=20476' 2024-05-08T00:34:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=20476' 2024-05-08T00:34:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=20476' '!=' 'object_number=73, tag_tid=11, entry_tid=20293' 2024-05-08T00:34:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=20476' = 'object_number=73, tag_tid=11, entry_tid=20293' 2024-05-08T00:34:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=20476' 2024-05-08T00:34:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=20476' 2024-05-08T00:34:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=20476' 2024-05-08T00:34:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=20476 2024-05-08T00:34:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 20476 2024-05-08T00:34:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=20640], [object_number=75, tag_tid=11, entry_tid=20639], [object_number=74, tag_tid=11, entry_tid=20638], [object_number=73, tag_tid=11, entry_tid=20637]]], state=connected] 2024-05-08T00:34:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=20640], [object_number=75, tag_tid=11, entry_tid=20639], [object_number=74, tag_tid=11, entry_tid=20638], [object_number=73, tag_tid=11, entry_tid=20637]]], state=connected] 2024-05-08T00:34:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=20640' 2024-05-08T00:34:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=20640' 2024-05-08T00:34:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=20640' '!=' 'object_number=72, tag_tid=11, entry_tid=20476' 2024-05-08T00:34:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=20640' = 'object_number=72, tag_tid=11, entry_tid=20476' 2024-05-08T00:34:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=20640' 2024-05-08T00:34:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=20640' 2024-05-08T00:34:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=20640' 2024-05-08T00:34:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=20640 2024-05-08T00:34:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 20640 2024-05-08T00:34:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=20795], [object_number=74, tag_tid=11, entry_tid=20794], [object_number=73, tag_tid=11, entry_tid=20793], [object_number=72, tag_tid=11, entry_tid=20792]]], state=connected] 2024-05-08T00:34:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=75, tag_tid=11, entry_tid=20795], [object_number=74, tag_tid=11, entry_tid=20794], [object_number=73, tag_tid=11, entry_tid=20793], [object_number=72, tag_tid=11, entry_tid=20792]]], state=connected] 2024-05-08T00:34:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=75, tag_tid=11, entry_tid=20795' 2024-05-08T00:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=75, tag_tid=11, entry_tid=20795' 2024-05-08T00:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=20795' '!=' 'object_number=72, tag_tid=11, entry_tid=20640' 2024-05-08T00:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=75, tag_tid=11, entry_tid=20795' = 'object_number=72, tag_tid=11, entry_tid=20640' 2024-05-08T00:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=75, tag_tid=11, entry_tid=20795' 2024-05-08T00:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=20795' 2024-05-08T00:34:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=75, tag_tid=11, entry_tid=20795' 2024-05-08T00:34:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=20795 2024-05-08T00:34:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 20795 2024-05-08T00:34:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=20908], [object_number=75, tag_tid=11, entry_tid=20907], [object_number=74, tag_tid=11, entry_tid=20906], [object_number=73, tag_tid=11, entry_tid=20905]]], state=connected] 2024-05-08T00:34:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=20908], [object_number=75, tag_tid=11, entry_tid=20907], [object_number=74, tag_tid=11, entry_tid=20906], [object_number=73, tag_tid=11, entry_tid=20905]]], state=connected] 2024-05-08T00:34:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=20908' 2024-05-08T00:34:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=20908' 2024-05-08T00:34:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=20908' '!=' 'object_number=75, tag_tid=11, entry_tid=20795' 2024-05-08T00:34:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=20908' = 'object_number=75, tag_tid=11, entry_tid=20795' 2024-05-08T00:34:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=20908' 2024-05-08T00:34:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=20908' 2024-05-08T00:34:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=20908' 2024-05-08T00:34:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=20908 2024-05-08T00:34:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 20908 2024-05-08T00:34:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=21029], [object_number=72, tag_tid=11, entry_tid=21028], [object_number=75, tag_tid=11, entry_tid=21027], [object_number=74, tag_tid=11, entry_tid=21026]]], state=connected] 2024-05-08T00:34:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=73, tag_tid=11, entry_tid=21029], [object_number=72, tag_tid=11, entry_tid=21028], [object_number=75, tag_tid=11, entry_tid=21027], [object_number=74, tag_tid=11, entry_tid=21026]]], state=connected] 2024-05-08T00:34:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=73, tag_tid=11, entry_tid=21029' 2024-05-08T00:34:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=11, entry_tid=21029' 2024-05-08T00:34:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=21029' '!=' 'object_number=72, tag_tid=11, entry_tid=20908' 2024-05-08T00:34:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=73, tag_tid=11, entry_tid=21029' = 'object_number=72, tag_tid=11, entry_tid=20908' 2024-05-08T00:34:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=73, tag_tid=11, entry_tid=21029' 2024-05-08T00:34:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=21029' 2024-05-08T00:34:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=11, entry_tid=21029' 2024-05-08T00:34:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=21029 2024-05-08T00:34:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 21029 2024-05-08T00:34:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=21154], [object_number=73, tag_tid=11, entry_tid=21153], [object_number=72, tag_tid=11, entry_tid=21152], [object_number=75, tag_tid=11, entry_tid=21151]]], state=connected] 2024-05-08T00:34:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=21154], [object_number=73, tag_tid=11, entry_tid=21153], [object_number=72, tag_tid=11, entry_tid=21152], [object_number=75, tag_tid=11, entry_tid=21151]]], state=connected] 2024-05-08T00:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=21154' 2024-05-08T00:34:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=21154' 2024-05-08T00:34:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=21154' '!=' 'object_number=73, tag_tid=11, entry_tid=21029' 2024-05-08T00:34:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=21154' = 'object_number=73, tag_tid=11, entry_tid=21029' 2024-05-08T00:34:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=21154' 2024-05-08T00:34:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=21154' 2024-05-08T00:34:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=21154' 2024-05-08T00:34:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=21154 2024-05-08T00:34:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 21154 2024-05-08T00:34:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=21306], [object_number=73, tag_tid=11, entry_tid=21305], [object_number=72, tag_tid=11, entry_tid=21304], [object_number=75, tag_tid=11, entry_tid=21303]]], state=connected] 2024-05-08T00:34:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=74, tag_tid=11, entry_tid=21306], [object_number=73, tag_tid=11, entry_tid=21305], [object_number=72, tag_tid=11, entry_tid=21304], [object_number=75, tag_tid=11, entry_tid=21303]]], state=connected] 2024-05-08T00:34:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=74, tag_tid=11, entry_tid=21306' 2024-05-08T00:34:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=74, tag_tid=11, entry_tid=21306' 2024-05-08T00:34:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=21306' '!=' 'object_number=74, tag_tid=11, entry_tid=21154' 2024-05-08T00:34:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=74, tag_tid=11, entry_tid=21306' = 'object_number=74, tag_tid=11, entry_tid=21154' 2024-05-08T00:34:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=74, tag_tid=11, entry_tid=21306' 2024-05-08T00:34:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=21306' 2024-05-08T00:34:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=74, tag_tid=11, entry_tid=21306' 2024-05-08T00:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=21306 2024-05-08T00:34:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 21306 2024-05-08T00:34:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=21320], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318], [object_number=73, tag_tid=11, entry_tid=21317]]], state=connected] 2024-05-08T00:34:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=21320], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318], [object_number=73, tag_tid=11, entry_tid=21317]]], state=connected] 2024-05-08T00:34:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=21320' 2024-05-08T00:34:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=21320' 2024-05-08T00:34:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=21320' '!=' 'object_number=74, tag_tid=11, entry_tid=21306' 2024-05-08T00:34:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:34:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=21320' = 'object_number=74, tag_tid=11, entry_tid=21306' 2024-05-08T00:34:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=72, tag_tid=11, entry_tid=21320' 2024-05-08T00:34:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=12 2024-05-08T00:34:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=21320' 2024-05-08T00:34:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:34:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=11 2024-05-08T00:34:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:34:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=72, tag_tid=11, entry_tid=21320' 2024-05-08T00:34:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:34:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:34:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=21320 2024-05-08T00:34:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 12 = 11 -a 1 -le 21320 2024-05-08T00:34:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:34:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:34:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=21320], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318], [object_number=73, tag_tid=11, entry_tid=21317]]], state=connected] 2024-05-08T00:34:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=72, tag_tid=11, entry_tid=21320], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318], [object_number=73, tag_tid=11, entry_tid=21317]]], state=connected] 2024-05-08T00:34:45.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=72, tag_tid=11, entry_tid=21320' 2024-05-08T00:34:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=72, tag_tid=11, entry_tid=21320' 2024-05-08T00:34:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=72, tag_tid=11, entry_tid=21320' '!=' 'object_number=72, tag_tid=11, entry_tid=21320' 2024-05-08T00:34:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:34:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.4 2024-05-08T00:34:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:34:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:34:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:34:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:34:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:34:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:34:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:34:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:34:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:34:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:34:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:34:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:34:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 18 2024-05-08T00:34:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 18 2024-05-08T00:34:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:34:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=73, tag_tid=12, entry_tid=1], [object_number=72, tag_tid=12, entry_tid=0], [object_number=75, tag_tid=11, entry_tid=21319], [object_number=74, tag_tid=11, entry_tid=21318]]], state=connected] 2024-05-08T00:34:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=73, tag_tid=12, entry_tid=1' -a 'object_number=73, tag_tid=12, entry_tid=1' = 'object_number=73, tag_tid=12, entry_tid=1' 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap4 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:34:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:34:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:34:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 7 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:34:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:34:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:32:54 2024 2024-05-08T00:34:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:34:31 2024 2024-05-08T00:34:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:34:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:34:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:34:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:34:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:34:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n snap3-clean ']' 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap3-clean 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap3-clean 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:34:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:34:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:47.773 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:34:47.774+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:34:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:34:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap4 2024-05-08T00:34:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap4 2024-05-08T00:34:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:34:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:34:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:34:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:34:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:48.935 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:34:48.936+0000 7f2ed9a0a640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:34:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:34:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:34:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:34:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap4-clean 2024-05-08T00:34:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-05-08T00:34:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:34:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap4-clean 2024-05-08T00:34:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:34:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:34:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:34:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:34:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:34:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:34:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:34:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=12 2024-05-08T00:34:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:34:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:34:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 62445568 bytes 2024-05-08T00:34:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 109036544 bytes 2024-05-08T00:34:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 172683264 bytes 2024-05-08T00:34:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 254333952 bytes 2024-05-08T00:35:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 270528512 bytes 2024-05-08T00:35:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 305494016 bytes 2024-05-08T00:35:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:35:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:35:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:35:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:35:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:35:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:35:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:35:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:35:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":4339405.2,"entries_behind_primary":0,"entries_per_second":271.9,"non_primary_position":{"entry_tid":283,"object_number":79,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":73,"tag_tid":13}} 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:34:56 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:35:01' 2024-05-08T00:35:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:35:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:35:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:35:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":4339405.2,"entries_behind_primary":0,"entries_per_second":271.9,"non_primary_position":{"entry_tid":283,"object_number":79,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":73,"tag_tid":13}} 2024-05-08T00:35:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:35:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:35:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:35:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:35:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:35:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:35:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:35:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:35:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:34:56' 2024-05-08T00:35:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:34:56' 2024-05-08T00:35:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:35:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:34:56' 2024-05-08T00:35:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:35:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:35:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:35:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":4339405.2,"entries_behind_primary":0,"entries_per_second":271.9,"non_primary_position":{"entry_tid":283,"object_number":79,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":73,"tag_tid":13}}42594266mirror.0smithi1162024-05-08 00:34:56cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:35:01' 2024-05-08T00:35:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:35:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:35:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:34:56' 2024-05-08T00:35:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:35:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:35:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:35:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:35:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:35:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap5 2024-05-08T00:35:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap5 2024-05-08T00:35:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:35:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap5 2024-05-08T00:35:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:35:15.564 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:15.562+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:35:15.564 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:15.562+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:35:15.565 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:15.562+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22000 2024-05-08T00:35:15.565 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:15.562+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:35:15.565 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:15.563+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:35:15.565 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:15.563+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:35:19.468 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:19.468+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:35:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:35:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:35:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:35:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:35:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:35:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:35:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:35:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:35:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:35:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:35:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:35:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:35:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:35:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:35:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:35:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:35:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:35:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:35:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:35:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:35:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=14901], [object_number=92, tag_tid=14, entry_tid=14900], [object_number=95, tag_tid=14, entry_tid=14899], [object_number=94, tag_tid=14, entry_tid=14898]]], state=connected] 2024-05-08T00:35:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=14901], [object_number=92, tag_tid=14, entry_tid=14900], [object_number=95, tag_tid=14, entry_tid=14899], [object_number=94, tag_tid=14, entry_tid=14898]]], state=connected] 2024-05-08T00:35:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=14901' 2024-05-08T00:35:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=14901' 2024-05-08T00:35:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=14901' '!=' '' 2024-05-08T00:35:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=14901' = '' 2024-05-08T00:35:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=14901' 2024-05-08T00:35:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=14901' 2024-05-08T00:35:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=14901' 2024-05-08T00:35:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14901 2024-05-08T00:35:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 14901 2024-05-08T00:35:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=15064], [object_number=95, tag_tid=14, entry_tid=15063], [object_number=94, tag_tid=14, entry_tid=15062], [object_number=93, tag_tid=14, entry_tid=15061]]], state=connected] 2024-05-08T00:35:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=15064], [object_number=95, tag_tid=14, entry_tid=15063], [object_number=94, tag_tid=14, entry_tid=15062], [object_number=93, tag_tid=14, entry_tid=15061]]], state=connected] 2024-05-08T00:35:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=15064' 2024-05-08T00:35:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=15064' 2024-05-08T00:35:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=15064' '!=' 'object_number=93, tag_tid=14, entry_tid=14901' 2024-05-08T00:35:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=15064' = 'object_number=93, tag_tid=14, entry_tid=14901' 2024-05-08T00:35:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=15064' 2024-05-08T00:35:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=15064' 2024-05-08T00:35:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=15064' 2024-05-08T00:35:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15064 2024-05-08T00:35:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 15064 2024-05-08T00:35:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=15244], [object_number=95, tag_tid=14, entry_tid=15243], [object_number=94, tag_tid=14, entry_tid=15242], [object_number=93, tag_tid=14, entry_tid=15241]]], state=connected] 2024-05-08T00:35:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=15244], [object_number=95, tag_tid=14, entry_tid=15243], [object_number=94, tag_tid=14, entry_tid=15242], [object_number=93, tag_tid=14, entry_tid=15241]]], state=connected] 2024-05-08T00:35:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=15244' 2024-05-08T00:35:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=15244' 2024-05-08T00:35:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=15244' '!=' 'object_number=92, tag_tid=14, entry_tid=15064' 2024-05-08T00:35:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=15244' = 'object_number=92, tag_tid=14, entry_tid=15064' 2024-05-08T00:35:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=15244' 2024-05-08T00:35:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=15244' 2024-05-08T00:35:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=15244' 2024-05-08T00:35:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15244 2024-05-08T00:35:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 15244 2024-05-08T00:35:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=15424], [object_number=95, tag_tid=14, entry_tid=15423], [object_number=94, tag_tid=14, entry_tid=15422], [object_number=93, tag_tid=14, entry_tid=15421]]], state=connected] 2024-05-08T00:35:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=15424], [object_number=95, tag_tid=14, entry_tid=15423], [object_number=94, tag_tid=14, entry_tid=15422], [object_number=93, tag_tid=14, entry_tid=15421]]], state=connected] 2024-05-08T00:35:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=15424' 2024-05-08T00:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=15424' 2024-05-08T00:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=15424' '!=' 'object_number=92, tag_tid=14, entry_tid=15244' 2024-05-08T00:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=15424' = 'object_number=92, tag_tid=14, entry_tid=15244' 2024-05-08T00:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=15424' 2024-05-08T00:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=15424' 2024-05-08T00:35:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=15424' 2024-05-08T00:35:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15424 2024-05-08T00:35:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 15424 2024-05-08T00:35:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=15612], [object_number=95, tag_tid=14, entry_tid=15611], [object_number=94, tag_tid=14, entry_tid=15610], [object_number=93, tag_tid=14, entry_tid=15609]]], state=connected] 2024-05-08T00:35:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=15612], [object_number=95, tag_tid=14, entry_tid=15611], [object_number=94, tag_tid=14, entry_tid=15610], [object_number=93, tag_tid=14, entry_tid=15609]]], state=connected] 2024-05-08T00:35:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=15612' 2024-05-08T00:35:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=15612' 2024-05-08T00:35:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=15612' '!=' 'object_number=92, tag_tid=14, entry_tid=15424' 2024-05-08T00:35:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=15612' = 'object_number=92, tag_tid=14, entry_tid=15424' 2024-05-08T00:35:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=15612' 2024-05-08T00:35:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=15612' 2024-05-08T00:35:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=15612' 2024-05-08T00:35:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15612 2024-05-08T00:35:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 15612 2024-05-08T00:35:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=15785], [object_number=92, tag_tid=14, entry_tid=15784], [object_number=95, tag_tid=14, entry_tid=15783], [object_number=94, tag_tid=14, entry_tid=15782]]], state=connected] 2024-05-08T00:35:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.929 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:36.929+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a270 2024-05-08T00:35:36.929 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:36.929+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:35:36.929 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:36.929+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:35:36.929 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:36.929+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22000 2024-05-08T00:35:36.929 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:36.929+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:35:36.930 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:36.931+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:35:36.930 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:36.931+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:35:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=15785], [object_number=92, tag_tid=14, entry_tid=15784], [object_number=95, tag_tid=14, entry_tid=15783], [object_number=94, tag_tid=14, entry_tid=15782]]], state=connected] 2024-05-08T00:35:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=15785' 2024-05-08T00:35:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=15785' 2024-05-08T00:35:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=15785' '!=' 'object_number=92, tag_tid=14, entry_tid=15612' 2024-05-08T00:35:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=15785' = 'object_number=92, tag_tid=14, entry_tid=15612' 2024-05-08T00:35:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=15785' 2024-05-08T00:35:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=15785' 2024-05-08T00:35:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=15785' 2024-05-08T00:35:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15785 2024-05-08T00:35:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 15785 2024-05-08T00:35:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=15943], [object_number=94, tag_tid=14, entry_tid=15942], [object_number=93, tag_tid=14, entry_tid=15941], [object_number=92, tag_tid=14, entry_tid=15940]]], state=connected] 2024-05-08T00:35:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=15943], [object_number=94, tag_tid=14, entry_tid=15942], [object_number=93, tag_tid=14, entry_tid=15941], [object_number=92, tag_tid=14, entry_tid=15940]]], state=connected] 2024-05-08T00:35:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=95, tag_tid=14, entry_tid=15943' 2024-05-08T00:35:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=95, tag_tid=14, entry_tid=15943' 2024-05-08T00:35:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=15943' '!=' 'object_number=93, tag_tid=14, entry_tid=15785' 2024-05-08T00:35:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=15943' = 'object_number=93, tag_tid=14, entry_tid=15785' 2024-05-08T00:35:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=95, tag_tid=14, entry_tid=15943' 2024-05-08T00:35:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=15943' 2024-05-08T00:35:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=15943' 2024-05-08T00:35:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15943 2024-05-08T00:35:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 15943 2024-05-08T00:35:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=16143], [object_number=94, tag_tid=14, entry_tid=16142], [object_number=93, tag_tid=14, entry_tid=16141], [object_number=92, tag_tid=14, entry_tid=16140]]], state=connected] 2024-05-08T00:35:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=16143], [object_number=94, tag_tid=14, entry_tid=16142], [object_number=93, tag_tid=14, entry_tid=16141], [object_number=92, tag_tid=14, entry_tid=16140]]], state=connected] 2024-05-08T00:35:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=95, tag_tid=14, entry_tid=16143' 2024-05-08T00:35:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=95, tag_tid=14, entry_tid=16143' 2024-05-08T00:35:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=16143' '!=' 'object_number=95, tag_tid=14, entry_tid=15943' 2024-05-08T00:35:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=16143' = 'object_number=95, tag_tid=14, entry_tid=15943' 2024-05-08T00:35:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=95, tag_tid=14, entry_tid=16143' 2024-05-08T00:35:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=16143' 2024-05-08T00:35:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=16143' 2024-05-08T00:35:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16143 2024-05-08T00:35:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 16143 2024-05-08T00:35:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=16316], [object_number=95, tag_tid=14, entry_tid=16315], [object_number=94, tag_tid=14, entry_tid=16314], [object_number=93, tag_tid=14, entry_tid=16313]]], state=connected] 2024-05-08T00:35:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=16316], [object_number=95, tag_tid=14, entry_tid=16315], [object_number=94, tag_tid=14, entry_tid=16314], [object_number=93, tag_tid=14, entry_tid=16313]]], state=connected] 2024-05-08T00:35:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=16316' 2024-05-08T00:35:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=16316' 2024-05-08T00:35:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=16316' '!=' 'object_number=95, tag_tid=14, entry_tid=16143' 2024-05-08T00:35:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=16316' = 'object_number=95, tag_tid=14, entry_tid=16143' 2024-05-08T00:35:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=16316' 2024-05-08T00:35:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=16316' 2024-05-08T00:35:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=16316' 2024-05-08T00:35:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16316 2024-05-08T00:35:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 16316 2024-05-08T00:35:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=16473], [object_number=92, tag_tid=14, entry_tid=16472], [object_number=95, tag_tid=14, entry_tid=16471], [object_number=94, tag_tid=14, entry_tid=16470]]], state=connected] 2024-05-08T00:35:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=16473], [object_number=92, tag_tid=14, entry_tid=16472], [object_number=95, tag_tid=14, entry_tid=16471], [object_number=94, tag_tid=14, entry_tid=16470]]], state=connected] 2024-05-08T00:35:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=16473' 2024-05-08T00:35:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=16473' 2024-05-08T00:35:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=16473' '!=' 'object_number=92, tag_tid=14, entry_tid=16316' 2024-05-08T00:35:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=16473' = 'object_number=92, tag_tid=14, entry_tid=16316' 2024-05-08T00:35:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=16473' 2024-05-08T00:35:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=16473' 2024-05-08T00:35:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=16473' 2024-05-08T00:35:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16473 2024-05-08T00:35:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 16473 2024-05-08T00:35:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=16619], [object_number=94, tag_tid=14, entry_tid=16618], [object_number=93, tag_tid=14, entry_tid=16617], [object_number=92, tag_tid=14, entry_tid=16616]]], state=connected] 2024-05-08T00:35:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=95, tag_tid=14, entry_tid=16619], [object_number=94, tag_tid=14, entry_tid=16618], [object_number=93, tag_tid=14, entry_tid=16617], [object_number=92, tag_tid=14, entry_tid=16616]]], state=connected] 2024-05-08T00:35:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=95, tag_tid=14, entry_tid=16619' 2024-05-08T00:35:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=95, tag_tid=14, entry_tid=16619' 2024-05-08T00:35:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=16619' '!=' 'object_number=93, tag_tid=14, entry_tid=16473' 2024-05-08T00:35:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=95, tag_tid=14, entry_tid=16619' = 'object_number=93, tag_tid=14, entry_tid=16473' 2024-05-08T00:35:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=95, tag_tid=14, entry_tid=16619' 2024-05-08T00:35:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=16619' 2024-05-08T00:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=95, tag_tid=14, entry_tid=16619' 2024-05-08T00:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16619 2024-05-08T00:35:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 16619 2024-05-08T00:35:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=16784], [object_number=95, tag_tid=14, entry_tid=16783], [object_number=94, tag_tid=14, entry_tid=16782], [object_number=93, tag_tid=14, entry_tid=16781]]], state=connected] 2024-05-08T00:35:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=16784], [object_number=95, tag_tid=14, entry_tid=16783], [object_number=94, tag_tid=14, entry_tid=16782], [object_number=93, tag_tid=14, entry_tid=16781]]], state=connected] 2024-05-08T00:35:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=16784' 2024-05-08T00:35:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=16784' 2024-05-08T00:35:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=16784' '!=' 'object_number=95, tag_tid=14, entry_tid=16619' 2024-05-08T00:35:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=16784' = 'object_number=95, tag_tid=14, entry_tid=16619' 2024-05-08T00:35:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=16784' 2024-05-08T00:35:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=16784' 2024-05-08T00:35:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=16784' 2024-05-08T00:35:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16784 2024-05-08T00:35:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 16784 2024-05-08T00:35:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=16972], [object_number=95, tag_tid=14, entry_tid=16971], [object_number=94, tag_tid=14, entry_tid=16970], [object_number=93, tag_tid=14, entry_tid=16969]]], state=connected] 2024-05-08T00:35:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=16972], [object_number=95, tag_tid=14, entry_tid=16971], [object_number=94, tag_tid=14, entry_tid=16970], [object_number=93, tag_tid=14, entry_tid=16969]]], state=connected] 2024-05-08T00:35:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=16972' 2024-05-08T00:35:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=16972' 2024-05-08T00:35:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=16972' '!=' 'object_number=92, tag_tid=14, entry_tid=16784' 2024-05-08T00:35:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=16972' = 'object_number=92, tag_tid=14, entry_tid=16784' 2024-05-08T00:35:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=16972' 2024-05-08T00:35:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=16972' 2024-05-08T00:35:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=16972' 2024-05-08T00:35:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16972 2024-05-08T00:35:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 16972 2024-05-08T00:35:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=17132], [object_number=95, tag_tid=14, entry_tid=17131], [object_number=94, tag_tid=14, entry_tid=17130], [object_number=93, tag_tid=14, entry_tid=17129]]], state=connected] 2024-05-08T00:35:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=92, tag_tid=14, entry_tid=17132], [object_number=95, tag_tid=14, entry_tid=17131], [object_number=94, tag_tid=14, entry_tid=17130], [object_number=93, tag_tid=14, entry_tid=17129]]], state=connected] 2024-05-08T00:35:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=92, tag_tid=14, entry_tid=17132' 2024-05-08T00:35:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=92, tag_tid=14, entry_tid=17132' 2024-05-08T00:35:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=17132' '!=' 'object_number=92, tag_tid=14, entry_tid=16972' 2024-05-08T00:35:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=92, tag_tid=14, entry_tid=17132' = 'object_number=92, tag_tid=14, entry_tid=16972' 2024-05-08T00:35:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=92, tag_tid=14, entry_tid=17132' 2024-05-08T00:35:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=17132' 2024-05-08T00:35:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=92, tag_tid=14, entry_tid=17132' 2024-05-08T00:35:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17132 2024-05-08T00:35:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 17132 2024-05-08T00:35:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=17306], [object_number=93, tag_tid=14, entry_tid=17305], [object_number=92, tag_tid=14, entry_tid=17304], [object_number=95, tag_tid=14, entry_tid=17303]]], state=connected] 2024-05-08T00:35:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=94, tag_tid=14, entry_tid=17306], [object_number=93, tag_tid=14, entry_tid=17305], [object_number=92, tag_tid=14, entry_tid=17304], [object_number=95, tag_tid=14, entry_tid=17303]]], state=connected] 2024-05-08T00:35:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=94, tag_tid=14, entry_tid=17306' 2024-05-08T00:35:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=94, tag_tid=14, entry_tid=17306' 2024-05-08T00:35:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=17306' '!=' 'object_number=92, tag_tid=14, entry_tid=17132' 2024-05-08T00:35:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=94, tag_tid=14, entry_tid=17306' = 'object_number=92, tag_tid=14, entry_tid=17132' 2024-05-08T00:35:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=94, tag_tid=14, entry_tid=17306' 2024-05-08T00:35:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=17306' 2024-05-08T00:35:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=94, tag_tid=14, entry_tid=17306' 2024-05-08T00:35:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17306 2024-05-08T00:35:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 17306 2024-05-08T00:35:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17473], [object_number=92, tag_tid=14, entry_tid=17472], [object_number=95, tag_tid=14, entry_tid=17471], [object_number=94, tag_tid=14, entry_tid=17470]]], state=connected] 2024-05-08T00:35:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 23 2024-05-08T00:35:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=93, tag_tid=14, entry_tid=17473], [object_number=92, tag_tid=14, entry_tid=17472], [object_number=95, tag_tid=14, entry_tid=17471], [object_number=94, tag_tid=14, entry_tid=17470]]], state=connected] 2024-05-08T00:35:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=93, tag_tid=14, entry_tid=17473' 2024-05-08T00:35:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=93, tag_tid=14, entry_tid=17473' 2024-05-08T00:35:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17473' '!=' 'object_number=94, tag_tid=14, entry_tid=17306' 2024-05-08T00:35:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=93, tag_tid=14, entry_tid=17473' = 'object_number=94, tag_tid=14, entry_tid=17306' 2024-05-08T00:35:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=93, tag_tid=14, entry_tid=17473' 2024-05-08T00:35:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17473' 2024-05-08T00:35:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=93, tag_tid=14, entry_tid=17473' 2024-05-08T00:35:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17473 2024-05-08T00:35:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 17473 2024-05-08T00:35:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:41.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=96, tag_tid=14, entry_tid=17592], [object_number=99, tag_tid=14, entry_tid=17591], [object_number=98, tag_tid=14, entry_tid=17590], [object_number=97, tag_tid=14, entry_tid=17589]]], state=connected] 2024-05-08T00:35:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=96, tag_tid=14, entry_tid=17592], [object_number=99, tag_tid=14, entry_tid=17591], [object_number=98, tag_tid=14, entry_tid=17590], [object_number=97, tag_tid=14, entry_tid=17589]]], state=connected] 2024-05-08T00:35:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=96, tag_tid=14, entry_tid=17592' 2024-05-08T00:35:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=96, tag_tid=14, entry_tid=17592' 2024-05-08T00:35:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=96, tag_tid=14, entry_tid=17592' '!=' 'object_number=93, tag_tid=14, entry_tid=17473' 2024-05-08T00:35:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=96, tag_tid=14, entry_tid=17592' = 'object_number=93, tag_tid=14, entry_tid=17473' 2024-05-08T00:35:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=96, tag_tid=14, entry_tid=17592' 2024-05-08T00:35:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=96, tag_tid=14, entry_tid=17592' 2024-05-08T00:35:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=96, tag_tid=14, entry_tid=17592' 2024-05-08T00:35:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17592 2024-05-08T00:35:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 17592 2024-05-08T00:35:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:41.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=17758], [object_number=97, tag_tid=14, entry_tid=17757], [object_number=96, tag_tid=14, entry_tid=17756], [object_number=99, tag_tid=14, entry_tid=17755]]], state=connected] 2024-05-08T00:35:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=17758], [object_number=97, tag_tid=14, entry_tid=17757], [object_number=96, tag_tid=14, entry_tid=17756], [object_number=99, tag_tid=14, entry_tid=17755]]], state=connected] 2024-05-08T00:35:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=98, tag_tid=14, entry_tid=17758' 2024-05-08T00:35:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=98, tag_tid=14, entry_tid=17758' 2024-05-08T00:35:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=98, tag_tid=14, entry_tid=17758' '!=' 'object_number=96, tag_tid=14, entry_tid=17592' 2024-05-08T00:35:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=98, tag_tid=14, entry_tid=17758' = 'object_number=96, tag_tid=14, entry_tid=17592' 2024-05-08T00:35:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=98, tag_tid=14, entry_tid=17758' 2024-05-08T00:35:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=98, tag_tid=14, entry_tid=17758' 2024-05-08T00:35:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=98, tag_tid=14, entry_tid=17758' 2024-05-08T00:35:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17758 2024-05-08T00:35:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 17758 2024-05-08T00:35:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=17910], [object_number=97, tag_tid=14, entry_tid=17909], [object_number=96, tag_tid=14, entry_tid=17908], [object_number=99, tag_tid=14, entry_tid=17907]]], state=connected] 2024-05-08T00:35:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=17910], [object_number=97, tag_tid=14, entry_tid=17909], [object_number=96, tag_tid=14, entry_tid=17908], [object_number=99, tag_tid=14, entry_tid=17907]]], state=connected] 2024-05-08T00:35:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=98, tag_tid=14, entry_tid=17910' 2024-05-08T00:35:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=98, tag_tid=14, entry_tid=17910' 2024-05-08T00:35:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=98, tag_tid=14, entry_tid=17910' '!=' 'object_number=98, tag_tid=14, entry_tid=17758' 2024-05-08T00:35:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=98, tag_tid=14, entry_tid=17910' = 'object_number=98, tag_tid=14, entry_tid=17758' 2024-05-08T00:35:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=98, tag_tid=14, entry_tid=17910' 2024-05-08T00:35:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=98, tag_tid=14, entry_tid=17910' 2024-05-08T00:35:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=98, tag_tid=14, entry_tid=17910' 2024-05-08T00:35:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=17910 2024-05-08T00:35:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 17910 2024-05-08T00:35:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=99, tag_tid=14, entry_tid=18059], [object_number=98, tag_tid=14, entry_tid=18058], [object_number=97, tag_tid=14, entry_tid=18057], [object_number=96, tag_tid=14, entry_tid=18056]]], state=connected] 2024-05-08T00:35:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=99, tag_tid=14, entry_tid=18059], [object_number=98, tag_tid=14, entry_tid=18058], [object_number=97, tag_tid=14, entry_tid=18057], [object_number=96, tag_tid=14, entry_tid=18056]]], state=connected] 2024-05-08T00:35:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=99, tag_tid=14, entry_tid=18059' 2024-05-08T00:35:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=99, tag_tid=14, entry_tid=18059' 2024-05-08T00:35:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=99, tag_tid=14, entry_tid=18059' '!=' 'object_number=98, tag_tid=14, entry_tid=17910' 2024-05-08T00:35:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=99, tag_tid=14, entry_tid=18059' = 'object_number=98, tag_tid=14, entry_tid=17910' 2024-05-08T00:35:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=99, tag_tid=14, entry_tid=18059' 2024-05-08T00:35:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=99, tag_tid=14, entry_tid=18059' 2024-05-08T00:35:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=99, tag_tid=14, entry_tid=18059' 2024-05-08T00:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18059 2024-05-08T00:35:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 18059 2024-05-08T00:35:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=97, tag_tid=14, entry_tid=18217], [object_number=96, tag_tid=14, entry_tid=18216], [object_number=99, tag_tid=14, entry_tid=18215], [object_number=98, tag_tid=14, entry_tid=18214]]], state=connected] 2024-05-08T00:35:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:43.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=97, tag_tid=14, entry_tid=18217], [object_number=96, tag_tid=14, entry_tid=18216], [object_number=99, tag_tid=14, entry_tid=18215], [object_number=98, tag_tid=14, entry_tid=18214]]], state=connected] 2024-05-08T00:35:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=97, tag_tid=14, entry_tid=18217' 2024-05-08T00:35:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=97, tag_tid=14, entry_tid=18217' 2024-05-08T00:35:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=97, tag_tid=14, entry_tid=18217' '!=' 'object_number=99, tag_tid=14, entry_tid=18059' 2024-05-08T00:35:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=97, tag_tid=14, entry_tid=18217' = 'object_number=99, tag_tid=14, entry_tid=18059' 2024-05-08T00:35:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=97, tag_tid=14, entry_tid=18217' 2024-05-08T00:35:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=14, entry_tid=18217' 2024-05-08T00:35:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=14, entry_tid=18217' 2024-05-08T00:35:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18217 2024-05-08T00:35:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 18217 2024-05-08T00:35:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=96, tag_tid=14, entry_tid=18364], [object_number=99, tag_tid=14, entry_tid=18363], [object_number=98, tag_tid=14, entry_tid=18362], [object_number=97, tag_tid=14, entry_tid=18361]]], state=connected] 2024-05-08T00:35:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=96, tag_tid=14, entry_tid=18364], [object_number=99, tag_tid=14, entry_tid=18363], [object_number=98, tag_tid=14, entry_tid=18362], [object_number=97, tag_tid=14, entry_tid=18361]]], state=connected] 2024-05-08T00:35:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=96, tag_tid=14, entry_tid=18364' 2024-05-08T00:35:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=96, tag_tid=14, entry_tid=18364' 2024-05-08T00:35:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=96, tag_tid=14, entry_tid=18364' '!=' 'object_number=97, tag_tid=14, entry_tid=18217' 2024-05-08T00:35:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=96, tag_tid=14, entry_tid=18364' = 'object_number=97, tag_tid=14, entry_tid=18217' 2024-05-08T00:35:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=96, tag_tid=14, entry_tid=18364' 2024-05-08T00:35:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=96, tag_tid=14, entry_tid=18364' 2024-05-08T00:35:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=96, tag_tid=14, entry_tid=18364' 2024-05-08T00:35:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18364 2024-05-08T00:35:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 18364 2024-05-08T00:35:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=99, tag_tid=14, entry_tid=18515], [object_number=98, tag_tid=14, entry_tid=18514], [object_number=97, tag_tid=14, entry_tid=18513], [object_number=96, tag_tid=14, entry_tid=18512]]], state=connected] 2024-05-08T00:35:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=99, tag_tid=14, entry_tid=18515], [object_number=98, tag_tid=14, entry_tid=18514], [object_number=97, tag_tid=14, entry_tid=18513], [object_number=96, tag_tid=14, entry_tid=18512]]], state=connected] 2024-05-08T00:35:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=99, tag_tid=14, entry_tid=18515' 2024-05-08T00:35:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=99, tag_tid=14, entry_tid=18515' 2024-05-08T00:35:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=99, tag_tid=14, entry_tid=18515' '!=' 'object_number=96, tag_tid=14, entry_tid=18364' 2024-05-08T00:35:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=99, tag_tid=14, entry_tid=18515' = 'object_number=96, tag_tid=14, entry_tid=18364' 2024-05-08T00:35:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=99, tag_tid=14, entry_tid=18515' 2024-05-08T00:35:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=99, tag_tid=14, entry_tid=18515' 2024-05-08T00:35:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=99, tag_tid=14, entry_tid=18515' 2024-05-08T00:35:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18515 2024-05-08T00:35:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 18515 2024-05-08T00:35:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=18774], [object_number=97, tag_tid=14, entry_tid=18773], [object_number=96, tag_tid=14, entry_tid=18772], [object_number=99, tag_tid=14, entry_tid=18771]]], state=connected] 2024-05-08T00:35:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=18774], [object_number=97, tag_tid=14, entry_tid=18773], [object_number=96, tag_tid=14, entry_tid=18772], [object_number=99, tag_tid=14, entry_tid=18771]]], state=connected] 2024-05-08T00:35:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=98, tag_tid=14, entry_tid=18774' 2024-05-08T00:35:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=98, tag_tid=14, entry_tid=18774' 2024-05-08T00:35:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=98, tag_tid=14, entry_tid=18774' '!=' 'object_number=99, tag_tid=14, entry_tid=18515' 2024-05-08T00:35:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=98, tag_tid=14, entry_tid=18774' = 'object_number=99, tag_tid=14, entry_tid=18515' 2024-05-08T00:35:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=98, tag_tid=14, entry_tid=18774' 2024-05-08T00:35:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=98, tag_tid=14, entry_tid=18774' 2024-05-08T00:35:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=98, tag_tid=14, entry_tid=18774' 2024-05-08T00:35:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18774 2024-05-08T00:35:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 18774 2024-05-08T00:35:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=99, tag_tid=14, entry_tid=18935], [object_number=98, tag_tid=14, entry_tid=18934], [object_number=97, tag_tid=14, entry_tid=18933], [object_number=96, tag_tid=14, entry_tid=18932]]], state=connected] 2024-05-08T00:35:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=99, tag_tid=14, entry_tid=18935], [object_number=98, tag_tid=14, entry_tid=18934], [object_number=97, tag_tid=14, entry_tid=18933], [object_number=96, tag_tid=14, entry_tid=18932]]], state=connected] 2024-05-08T00:35:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=99, tag_tid=14, entry_tid=18935' 2024-05-08T00:35:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=99, tag_tid=14, entry_tid=18935' 2024-05-08T00:35:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=99, tag_tid=14, entry_tid=18935' '!=' 'object_number=98, tag_tid=14, entry_tid=18774' 2024-05-08T00:35:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=99, tag_tid=14, entry_tid=18935' = 'object_number=98, tag_tid=14, entry_tid=18774' 2024-05-08T00:35:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=99, tag_tid=14, entry_tid=18935' 2024-05-08T00:35:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=99, tag_tid=14, entry_tid=18935' 2024-05-08T00:35:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:45.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=99, tag_tid=14, entry_tid=18935' 2024-05-08T00:35:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=18935 2024-05-08T00:35:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 18935 2024-05-08T00:35:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=99, tag_tid=14, entry_tid=19095], [object_number=98, tag_tid=14, entry_tid=19094], [object_number=97, tag_tid=14, entry_tid=19093], [object_number=96, tag_tid=14, entry_tid=19092]]], state=connected] 2024-05-08T00:35:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=99, tag_tid=14, entry_tid=19095], [object_number=98, tag_tid=14, entry_tid=19094], [object_number=97, tag_tid=14, entry_tid=19093], [object_number=96, tag_tid=14, entry_tid=19092]]], state=connected] 2024-05-08T00:35:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=99, tag_tid=14, entry_tid=19095' 2024-05-08T00:35:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=99, tag_tid=14, entry_tid=19095' 2024-05-08T00:35:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=99, tag_tid=14, entry_tid=19095' '!=' 'object_number=99, tag_tid=14, entry_tid=18935' 2024-05-08T00:35:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=99, tag_tid=14, entry_tid=19095' = 'object_number=99, tag_tid=14, entry_tid=18935' 2024-05-08T00:35:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=99, tag_tid=14, entry_tid=19095' 2024-05-08T00:35:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=99, tag_tid=14, entry_tid=19095' 2024-05-08T00:35:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:45.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=99, tag_tid=14, entry_tid=19095' 2024-05-08T00:35:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=19095 2024-05-08T00:35:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 19095 2024-05-08T00:35:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=96, tag_tid=14, entry_tid=19292], [object_number=99, tag_tid=14, entry_tid=19291], [object_number=98, tag_tid=14, entry_tid=19290], [object_number=97, tag_tid=14, entry_tid=19289]]], state=connected] 2024-05-08T00:35:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=96, tag_tid=14, entry_tid=19292], [object_number=99, tag_tid=14, entry_tid=19291], [object_number=98, tag_tid=14, entry_tid=19290], [object_number=97, tag_tid=14, entry_tid=19289]]], state=connected] 2024-05-08T00:35:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=96, tag_tid=14, entry_tid=19292' 2024-05-08T00:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=96, tag_tid=14, entry_tid=19292' 2024-05-08T00:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=96, tag_tid=14, entry_tid=19292' '!=' 'object_number=99, tag_tid=14, entry_tid=19095' 2024-05-08T00:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=96, tag_tid=14, entry_tid=19292' = 'object_number=99, tag_tid=14, entry_tid=19095' 2024-05-08T00:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=96, tag_tid=14, entry_tid=19292' 2024-05-08T00:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=96, tag_tid=14, entry_tid=19292' 2024-05-08T00:35:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=96, tag_tid=14, entry_tid=19292' 2024-05-08T00:35:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=19292 2024-05-08T00:35:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 19292 2024-05-08T00:35:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=19298], [object_number=97, tag_tid=14, entry_tid=19297], [object_number=96, tag_tid=14, entry_tid=19296], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=19298], [object_number=97, tag_tid=14, entry_tid=19297], [object_number=96, tag_tid=14, entry_tid=19296], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=98, tag_tid=14, entry_tid=19298' 2024-05-08T00:35:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=98, tag_tid=14, entry_tid=19298' 2024-05-08T00:35:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=98, tag_tid=14, entry_tid=19298' '!=' 'object_number=96, tag_tid=14, entry_tid=19292' 2024-05-08T00:35:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:35:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=98, tag_tid=14, entry_tid=19298' = 'object_number=96, tag_tid=14, entry_tid=19292' 2024-05-08T00:35:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=98, tag_tid=14, entry_tid=19298' 2024-05-08T00:35:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=15 2024-05-08T00:35:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=98, tag_tid=14, entry_tid=19298' 2024-05-08T00:35:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:35:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=14 2024-05-08T00:35:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:35:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=98, tag_tid=14, entry_tid=19298' 2024-05-08T00:35:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:35:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:35:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=19298 2024-05-08T00:35:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 15 = 14 -a 1 -le 19298 2024-05-08T00:35:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:35:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:35:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=19298], [object_number=97, tag_tid=14, entry_tid=19297], [object_number=96, tag_tid=14, entry_tid=19296], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=98, tag_tid=14, entry_tid=19298], [object_number=97, tag_tid=14, entry_tid=19297], [object_number=96, tag_tid=14, entry_tid=19296], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=98, tag_tid=14, entry_tid=19298' 2024-05-08T00:35:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=98, tag_tid=14, entry_tid=19298' 2024-05-08T00:35:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=98, tag_tid=14, entry_tid=19298' '!=' 'object_number=98, tag_tid=14, entry_tid=19298' 2024-05-08T00:35:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:35:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.4 2024-05-08T00:35:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:35:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:35:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:35:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:35:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:35:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:35:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:35:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:35:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:35:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:35:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:35:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 24 2024-05-08T00:35:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 24 2024-05-08T00:35:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:35:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=97, tag_tid=15, entry_tid=1], [object_number=96, tag_tid=15, entry_tid=0], [object_number=98, tag_tid=14, entry_tid=19298], [object_number=99, tag_tid=14, entry_tid=19295]]], state=connected] 2024-05-08T00:35:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:35:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=97, tag_tid=15, entry_tid=1' -a 'object_number=97, tag_tid=15, entry_tid=1' = 'object_number=97, tag_tid=15, entry_tid=1' 2024-05-08T00:35:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-05-08T00:35:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:35:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap5 2024-05-08T00:35:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:35:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:35:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:35:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-05-08T00:35:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 9 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:34:47 2024 2024-05-08T00:35:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:35:32 2024 2024-05-08T00:35:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:35:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:35:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:35:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:35:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:35:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n snap4-clean ']' 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap4-clean 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap4-clean 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:35:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:35:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:49.499 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:35:49.500+0000 7f530a106640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-05-08T00:35:49.805 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:49.806+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:35:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:35:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap5 2024-05-08T00:35:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap5 2024-05-08T00:35:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:35:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:35:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:35:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:35:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:51.054 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:35:51.056+0000 7f2ed9209640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:35:51.262 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:51.263+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-05-08T00:35:51.263 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:51.264+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 7 2024-05-08T00:35:51.588 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:35:51.590+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:35:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:35:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:35:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:35:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap5-clean 2024-05-08T00:35:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-05-08T00:35:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:35:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap5-clean 2024-05-08T00:35:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:35:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:35:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:35:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:35:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:35:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:35:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:35:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=12 2024-05-08T00:35:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:35:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:35:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 23652352 bytes 2024-05-08T00:35:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 51205120 bytes 2024-05-08T00:35:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 82656256 bytes 2024-05-08T00:35:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 108151808 bytes 2024-05-08T00:36:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 180660224 bytes 2024-05-08T00:36:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 219881472 bytes 2024-05-08T00:36:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:700 IOs, 261056512 bytes 2024-05-08T00:36:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:36:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:36:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:36:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:36:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:36:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:36:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:36:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:36:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:36:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:36:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:36:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:36:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:36:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:36:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:36:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:36:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:36:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:36:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":5440405.03,"entries_behind_primary":0,"entries_per_second":339.17,"non_primary_position":{"entry_tid":1,"object_number":97,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":97,"tag_tid":15}} 2024-05-08T00:36:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:36:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:35:56 2024-05-08T00:36:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:36:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:36:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:36:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:36:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:36:01' 2024-05-08T00:36:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:36:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:36:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:36:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":5440405.03,"entries_behind_primary":0,"entries_per_second":339.17,"non_primary_position":{"entry_tid":1,"object_number":97,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":97,"tag_tid":15}} 2024-05-08T00:36:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:36:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:36:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:36:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:36:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:36:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:36:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:36:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:35:56' 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:35:56' 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:35:56' 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:36:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":5440405.03,"entries_behind_primary":0,"entries_per_second":339.17,"non_primary_position":{"entry_tid":1,"object_number":97,"tag_tid":15},"primary_position":{"entry_tid":1,"object_number":97,"tag_tid":15}}42594266mirror.0smithi1162024-05-08 00:35:56cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:36:01' 2024-05-08T00:36:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:36:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:36:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:35:56' 2024-05-08T00:36:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:36:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:36:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:36:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:36:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:36:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap6 2024-05-08T00:36:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap6 2024-05-08T00:36:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:36:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap6 2024-05-08T00:36:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:36:23.740 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:23.740+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:36:23.740 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:23.740+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:36:23.740 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:23.741+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:36:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:36:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:36:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:36:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:36:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:36:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:36:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:36:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:36:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:36:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:36:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:36:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:36:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:36:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:36:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:36:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:36:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:36:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:36:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:36:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:36:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:36:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:36:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:36:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:36:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:36:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:36:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:36:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:36:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:36:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=14406], [object_number=113, tag_tid=17, entry_tid=14405], [object_number=112, tag_tid=17, entry_tid=14404], [object_number=115, tag_tid=17, entry_tid=14403]]], state=connected] 2024-05-08T00:36:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=14406], [object_number=113, tag_tid=17, entry_tid=14405], [object_number=112, tag_tid=17, entry_tid=14404], [object_number=115, tag_tid=17, entry_tid=14403]]], state=connected] 2024-05-08T00:36:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=14406' 2024-05-08T00:36:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=14406' 2024-05-08T00:36:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=14406' '!=' '' 2024-05-08T00:36:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=14406' = '' 2024-05-08T00:36:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=114, tag_tid=17, entry_tid=14406' 2024-05-08T00:36:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=14406' 2024-05-08T00:36:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=14406' 2024-05-08T00:36:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14406 2024-05-08T00:36:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 14406 2024-05-08T00:36:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=14556], [object_number=115, tag_tid=17, entry_tid=14555], [object_number=114, tag_tid=17, entry_tid=14554], [object_number=113, tag_tid=17, entry_tid=14553]]], state=connected] 2024-05-08T00:36:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=14556], [object_number=115, tag_tid=17, entry_tid=14555], [object_number=114, tag_tid=17, entry_tid=14554], [object_number=113, tag_tid=17, entry_tid=14553]]], state=connected] 2024-05-08T00:36:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=112, tag_tid=17, entry_tid=14556' 2024-05-08T00:36:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=112, tag_tid=17, entry_tid=14556' 2024-05-08T00:36:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=14556' '!=' 'object_number=114, tag_tid=17, entry_tid=14406' 2024-05-08T00:36:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=14556' = 'object_number=114, tag_tid=17, entry_tid=14406' 2024-05-08T00:36:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=112, tag_tid=17, entry_tid=14556' 2024-05-08T00:36:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=14556' 2024-05-08T00:36:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=14556' 2024-05-08T00:36:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14556 2024-05-08T00:36:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 14556 2024-05-08T00:36:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=14743], [object_number=114, tag_tid=17, entry_tid=14742], [object_number=113, tag_tid=17, entry_tid=14741], [object_number=112, tag_tid=17, entry_tid=14740]]], state=connected] 2024-05-08T00:36:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=14743], [object_number=114, tag_tid=17, entry_tid=14742], [object_number=113, tag_tid=17, entry_tid=14741], [object_number=112, tag_tid=17, entry_tid=14740]]], state=connected] 2024-05-08T00:36:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=115, tag_tid=17, entry_tid=14743' 2024-05-08T00:36:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=115, tag_tid=17, entry_tid=14743' 2024-05-08T00:36:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=14743' '!=' 'object_number=112, tag_tid=17, entry_tid=14556' 2024-05-08T00:36:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=14743' = 'object_number=112, tag_tid=17, entry_tid=14556' 2024-05-08T00:36:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=115, tag_tid=17, entry_tid=14743' 2024-05-08T00:36:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=14743' 2024-05-08T00:36:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=14743' 2024-05-08T00:36:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14743 2024-05-08T00:36:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 14743 2024-05-08T00:36:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=14902], [object_number=113, tag_tid=17, entry_tid=14901], [object_number=112, tag_tid=17, entry_tid=14900], [object_number=115, tag_tid=17, entry_tid=14899]]], state=connected] 2024-05-08T00:36:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=114, tag_tid=17, entry_tid=14902], [object_number=113, tag_tid=17, entry_tid=14901], [object_number=112, tag_tid=17, entry_tid=14900], [object_number=115, tag_tid=17, entry_tid=14899]]], state=connected] 2024-05-08T00:36:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=114, tag_tid=17, entry_tid=14902' 2024-05-08T00:36:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=114, tag_tid=17, entry_tid=14902' 2024-05-08T00:36:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=14902' '!=' 'object_number=115, tag_tid=17, entry_tid=14743' 2024-05-08T00:36:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=114, tag_tid=17, entry_tid=14902' = 'object_number=115, tag_tid=17, entry_tid=14743' 2024-05-08T00:36:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=114, tag_tid=17, entry_tid=14902' 2024-05-08T00:36:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=14902' 2024-05-08T00:36:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=114, tag_tid=17, entry_tid=14902' 2024-05-08T00:36:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=14902 2024-05-08T00:36:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 14902 2024-05-08T00:36:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=15060], [object_number=115, tag_tid=17, entry_tid=15059], [object_number=114, tag_tid=17, entry_tid=15058], [object_number=113, tag_tid=17, entry_tid=15057]]], state=connected] 2024-05-08T00:36:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=112, tag_tid=17, entry_tid=15060], [object_number=115, tag_tid=17, entry_tid=15059], [object_number=114, tag_tid=17, entry_tid=15058], [object_number=113, tag_tid=17, entry_tid=15057]]], state=connected] 2024-05-08T00:36:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=112, tag_tid=17, entry_tid=15060' 2024-05-08T00:36:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=112, tag_tid=17, entry_tid=15060' 2024-05-08T00:36:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=15060' '!=' 'object_number=114, tag_tid=17, entry_tid=14902' 2024-05-08T00:36:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=112, tag_tid=17, entry_tid=15060' = 'object_number=114, tag_tid=17, entry_tid=14902' 2024-05-08T00:36:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=112, tag_tid=17, entry_tid=15060' 2024-05-08T00:36:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=15060' 2024-05-08T00:36:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=112, tag_tid=17, entry_tid=15060' 2024-05-08T00:36:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15060 2024-05-08T00:36:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 15060 2024-05-08T00:36:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=15207], [object_number=114, tag_tid=17, entry_tid=15206], [object_number=113, tag_tid=17, entry_tid=15205], [object_number=112, tag_tid=17, entry_tid=15204]]], state=connected] 2024-05-08T00:36:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=115, tag_tid=17, entry_tid=15207], [object_number=114, tag_tid=17, entry_tid=15206], [object_number=113, tag_tid=17, entry_tid=15205], [object_number=112, tag_tid=17, entry_tid=15204]]], state=connected] 2024-05-08T00:36:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=115, tag_tid=17, entry_tid=15207' 2024-05-08T00:36:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=115, tag_tid=17, entry_tid=15207' 2024-05-08T00:36:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=15207' '!=' 'object_number=112, tag_tid=17, entry_tid=15060' 2024-05-08T00:36:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=115, tag_tid=17, entry_tid=15207' = 'object_number=112, tag_tid=17, entry_tid=15060' 2024-05-08T00:36:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=115, tag_tid=17, entry_tid=15207' 2024-05-08T00:36:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=15207' 2024-05-08T00:36:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=115, tag_tid=17, entry_tid=15207' 2024-05-08T00:36:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15207 2024-05-08T00:36:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 15207 2024-05-08T00:36:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=15349], [object_number=116, tag_tid=17, entry_tid=15348], [object_number=115, tag_tid=17, entry_tid=15347], [object_number=114, tag_tid=17, entry_tid=15346]]], state=connected] 2024-05-08T00:36:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 28 2024-05-08T00:36:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=113, tag_tid=17, entry_tid=15349], [object_number=116, tag_tid=17, entry_tid=15348], [object_number=115, tag_tid=17, entry_tid=15347], [object_number=114, tag_tid=17, entry_tid=15346]]], state=connected] 2024-05-08T00:36:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=113, tag_tid=17, entry_tid=15349' 2024-05-08T00:36:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=113, tag_tid=17, entry_tid=15349' 2024-05-08T00:36:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=15349' '!=' 'object_number=115, tag_tid=17, entry_tid=15207' 2024-05-08T00:36:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=113, tag_tid=17, entry_tid=15349' = 'object_number=115, tag_tid=17, entry_tid=15207' 2024-05-08T00:36:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=113, tag_tid=17, entry_tid=15349' 2024-05-08T00:36:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=15349' 2024-05-08T00:36:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=113, tag_tid=17, entry_tid=15349' 2024-05-08T00:36:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15349 2024-05-08T00:36:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 15349 2024-05-08T00:36:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=118, tag_tid=17, entry_tid=15462], [object_number=117, tag_tid=17, entry_tid=15461], [object_number=116, tag_tid=17, entry_tid=15460], [object_number=119, tag_tid=17, entry_tid=15459]]], state=connected] 2024-05-08T00:36:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=118, tag_tid=17, entry_tid=15462], [object_number=117, tag_tid=17, entry_tid=15461], [object_number=116, tag_tid=17, entry_tid=15460], [object_number=119, tag_tid=17, entry_tid=15459]]], state=connected] 2024-05-08T00:36:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=118, tag_tid=17, entry_tid=15462' 2024-05-08T00:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=118, tag_tid=17, entry_tid=15462' 2024-05-08T00:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=118, tag_tid=17, entry_tid=15462' '!=' 'object_number=113, tag_tid=17, entry_tid=15349' 2024-05-08T00:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=118, tag_tid=17, entry_tid=15462' = 'object_number=113, tag_tid=17, entry_tid=15349' 2024-05-08T00:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=118, tag_tid=17, entry_tid=15462' 2024-05-08T00:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=118, tag_tid=17, entry_tid=15462' 2024-05-08T00:36:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=118, tag_tid=17, entry_tid=15462' 2024-05-08T00:36:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15462 2024-05-08T00:36:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 15462 2024-05-08T00:36:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:40.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=116, tag_tid=17, entry_tid=15688], [object_number=119, tag_tid=17, entry_tid=15687], [object_number=118, tag_tid=17, entry_tid=15686], [object_number=117, tag_tid=17, entry_tid=15685]]], state=connected] 2024-05-08T00:36:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:40.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:40.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=116, tag_tid=17, entry_tid=15688], [object_number=119, tag_tid=17, entry_tid=15687], [object_number=118, tag_tid=17, entry_tid=15686], [object_number=117, tag_tid=17, entry_tid=15685]]], state=connected] 2024-05-08T00:36:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=116, tag_tid=17, entry_tid=15688' 2024-05-08T00:36:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=116, tag_tid=17, entry_tid=15688' 2024-05-08T00:36:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=116, tag_tid=17, entry_tid=15688' '!=' 'object_number=118, tag_tid=17, entry_tid=15462' 2024-05-08T00:36:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=116, tag_tid=17, entry_tid=15688' = 'object_number=118, tag_tid=17, entry_tid=15462' 2024-05-08T00:36:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=116, tag_tid=17, entry_tid=15688' 2024-05-08T00:36:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=116, tag_tid=17, entry_tid=15688' 2024-05-08T00:36:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=116, tag_tid=17, entry_tid=15688' 2024-05-08T00:36:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15688 2024-05-08T00:36:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 15688 2024-05-08T00:36:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=118, tag_tid=17, entry_tid=15838], [object_number=117, tag_tid=17, entry_tid=15837], [object_number=116, tag_tid=17, entry_tid=15836], [object_number=119, tag_tid=17, entry_tid=15835]]], state=connected] 2024-05-08T00:36:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:41.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=118, tag_tid=17, entry_tid=15838], [object_number=117, tag_tid=17, entry_tid=15837], [object_number=116, tag_tid=17, entry_tid=15836], [object_number=119, tag_tid=17, entry_tid=15835]]], state=connected] 2024-05-08T00:36:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=118, tag_tid=17, entry_tid=15838' 2024-05-08T00:36:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=118, tag_tid=17, entry_tid=15838' 2024-05-08T00:36:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=118, tag_tid=17, entry_tid=15838' '!=' 'object_number=116, tag_tid=17, entry_tid=15688' 2024-05-08T00:36:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=118, tag_tid=17, entry_tid=15838' = 'object_number=116, tag_tid=17, entry_tid=15688' 2024-05-08T00:36:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=118, tag_tid=17, entry_tid=15838' 2024-05-08T00:36:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=118, tag_tid=17, entry_tid=15838' 2024-05-08T00:36:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=118, tag_tid=17, entry_tid=15838' 2024-05-08T00:36:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15838 2024-05-08T00:36:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 15838 2024-05-08T00:36:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=119, tag_tid=17, entry_tid=15983], [object_number=118, tag_tid=17, entry_tid=15982], [object_number=117, tag_tid=17, entry_tid=15981], [object_number=116, tag_tid=17, entry_tid=15980]]], state=connected] 2024-05-08T00:36:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=119, tag_tid=17, entry_tid=15983], [object_number=118, tag_tid=17, entry_tid=15982], [object_number=117, tag_tid=17, entry_tid=15981], [object_number=116, tag_tid=17, entry_tid=15980]]], state=connected] 2024-05-08T00:36:41.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=119, tag_tid=17, entry_tid=15983' 2024-05-08T00:36:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=119, tag_tid=17, entry_tid=15983' 2024-05-08T00:36:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=119, tag_tid=17, entry_tid=15983' '!=' 'object_number=118, tag_tid=17, entry_tid=15838' 2024-05-08T00:36:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=119, tag_tid=17, entry_tid=15983' = 'object_number=118, tag_tid=17, entry_tid=15838' 2024-05-08T00:36:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=119, tag_tid=17, entry_tid=15983' 2024-05-08T00:36:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=119, tag_tid=17, entry_tid=15983' 2024-05-08T00:36:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=119, tag_tid=17, entry_tid=15983' 2024-05-08T00:36:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15983 2024-05-08T00:36:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 15983 2024-05-08T00:36:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=117, tag_tid=17, entry_tid=16129], [object_number=116, tag_tid=17, entry_tid=16128], [object_number=119, tag_tid=17, entry_tid=16127], [object_number=118, tag_tid=17, entry_tid=16126]]], state=connected] 2024-05-08T00:36:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=117, tag_tid=17, entry_tid=16129], [object_number=116, tag_tid=17, entry_tid=16128], [object_number=119, tag_tid=17, entry_tid=16127], [object_number=118, tag_tid=17, entry_tid=16126]]], state=connected] 2024-05-08T00:36:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=117, tag_tid=17, entry_tid=16129' 2024-05-08T00:36:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=117, tag_tid=17, entry_tid=16129' 2024-05-08T00:36:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=117, tag_tid=17, entry_tid=16129' '!=' 'object_number=119, tag_tid=17, entry_tid=15983' 2024-05-08T00:36:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=117, tag_tid=17, entry_tid=16129' = 'object_number=119, tag_tid=17, entry_tid=15983' 2024-05-08T00:36:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=117, tag_tid=17, entry_tid=16129' 2024-05-08T00:36:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=17, entry_tid=16129' 2024-05-08T00:36:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=17, entry_tid=16129' 2024-05-08T00:36:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16129 2024-05-08T00:36:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 16129 2024-05-08T00:36:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=118, tag_tid=17, entry_tid=16302], [object_number=117, tag_tid=17, entry_tid=16301], [object_number=116, tag_tid=17, entry_tid=16300], [object_number=119, tag_tid=17, entry_tid=16299]]], state=connected] 2024-05-08T00:36:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=118, tag_tid=17, entry_tid=16302], [object_number=117, tag_tid=17, entry_tid=16301], [object_number=116, tag_tid=17, entry_tid=16300], [object_number=119, tag_tid=17, entry_tid=16299]]], state=connected] 2024-05-08T00:36:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=118, tag_tid=17, entry_tid=16302' 2024-05-08T00:36:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=118, tag_tid=17, entry_tid=16302' 2024-05-08T00:36:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=118, tag_tid=17, entry_tid=16302' '!=' 'object_number=117, tag_tid=17, entry_tid=16129' 2024-05-08T00:36:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=118, tag_tid=17, entry_tid=16302' = 'object_number=117, tag_tid=17, entry_tid=16129' 2024-05-08T00:36:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=118, tag_tid=17, entry_tid=16302' 2024-05-08T00:36:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=118, tag_tid=17, entry_tid=16302' 2024-05-08T00:36:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=118, tag_tid=17, entry_tid=16302' 2024-05-08T00:36:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16302 2024-05-08T00:36:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 16302 2024-05-08T00:36:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=119, tag_tid=17, entry_tid=16487], [object_number=118, tag_tid=17, entry_tid=16486], [object_number=117, tag_tid=17, entry_tid=16485], [object_number=116, tag_tid=17, entry_tid=16484]]], state=connected] 2024-05-08T00:36:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=119, tag_tid=17, entry_tid=16487], [object_number=118, tag_tid=17, entry_tid=16486], [object_number=117, tag_tid=17, entry_tid=16485], [object_number=116, tag_tid=17, entry_tid=16484]]], state=connected] 2024-05-08T00:36:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=119, tag_tid=17, entry_tid=16487' 2024-05-08T00:36:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=119, tag_tid=17, entry_tid=16487' 2024-05-08T00:36:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=119, tag_tid=17, entry_tid=16487' '!=' 'object_number=118, tag_tid=17, entry_tid=16302' 2024-05-08T00:36:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=119, tag_tid=17, entry_tid=16487' = 'object_number=118, tag_tid=17, entry_tid=16302' 2024-05-08T00:36:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=119, tag_tid=17, entry_tid=16487' 2024-05-08T00:36:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=119, tag_tid=17, entry_tid=16487' 2024-05-08T00:36:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=119, tag_tid=17, entry_tid=16487' 2024-05-08T00:36:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16487 2024-05-08T00:36:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 16487 2024-05-08T00:36:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=116, tag_tid=17, entry_tid=16632], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630], [object_number=117, tag_tid=17, entry_tid=16629]]], state=connected] 2024-05-08T00:36:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=116, tag_tid=17, entry_tid=16632], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630], [object_number=117, tag_tid=17, entry_tid=16629]]], state=connected] 2024-05-08T00:36:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=116, tag_tid=17, entry_tid=16632' 2024-05-08T00:36:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=116, tag_tid=17, entry_tid=16632' 2024-05-08T00:36:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=116, tag_tid=17, entry_tid=16632' '!=' 'object_number=119, tag_tid=17, entry_tid=16487' 2024-05-08T00:36:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:36:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=116, tag_tid=17, entry_tid=16632' = 'object_number=119, tag_tid=17, entry_tid=16487' 2024-05-08T00:36:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=116, tag_tid=17, entry_tid=16632' 2024-05-08T00:36:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=18 2024-05-08T00:36:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=116, tag_tid=17, entry_tid=16632' 2024-05-08T00:36:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:36:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=17 2024-05-08T00:36:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:36:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=116, tag_tid=17, entry_tid=16632' 2024-05-08T00:36:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:36:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:36:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=16632 2024-05-08T00:36:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 18 = 17 -a 1 -le 16632 2024-05-08T00:36:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:36:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:36:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:36:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:36:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:36:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:36:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:36:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:36:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:36:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:36:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:36:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:36:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:36:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:36:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:36:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:36:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:36:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:36:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:36:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:36:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:36:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:36:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:36:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 29 2024-05-08T00:36:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 29 2024-05-08T00:36:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:36:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=117, tag_tid=18, entry_tid=1], [object_number=116, tag_tid=18, entry_tid=0], [object_number=119, tag_tid=17, entry_tid=16631], [object_number=118, tag_tid=17, entry_tid=16630]]], state=connected] 2024-05-08T00:36:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:36:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=117, tag_tid=18, entry_tid=1' -a 'object_number=117, tag_tid=18, entry_tid=1' = 'object_number=117, tag_tid=18, entry_tid=1' 2024-05-08T00:36:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-05-08T00:36:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap6 2024-05-08T00:36:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:36:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:36:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:36:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 11 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:36:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:36:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:35:49 2024 2024-05-08T00:36:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:36:33 2024 2024-05-08T00:36:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:36:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:36:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:36:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:36:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:36:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:36:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n snap5-clean ']' 2024-05-08T00:36:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap5-clean 2024-05-08T00:36:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap5-clean 2024-05-08T00:36:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:36:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:36:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:36:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:36:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:45.393 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:45.394+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:36:45.727 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:45.728+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:36:45.727 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:45.728+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:36:45.728 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:45.729+0000 7fbaf69c6640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-05-08T00:36:45.908 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:45.909+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:36:46.041 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:46.042+0000 7fbaf69c6640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:36:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:36:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap6 2024-05-08T00:36:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap6 2024-05-08T00:36:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:36:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:36:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:36:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:36:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:47.325 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:47.326+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-05-08T00:36:47.525 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:47.526+0000 7fbafa1cd640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:36:47.525 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:47.526+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-05-08T00:36:47.525 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:36:47.526+0000 7fbaf99cc640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-05-08T00:36:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:36:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:36:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap6-clean 2024-05-08T00:36:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-05-08T00:36:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:36:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap6-clean 2024-05-08T00:36:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:36:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:36:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:36:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:36:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:36:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=7 2024-05-08T00:36:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:36:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:36:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 624640 bytes 2024-05-08T00:36:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 75497472 bytes 2024-05-08T00:36:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:36:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:36:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:36:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:36:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:36:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:36:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:36:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:36:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:36:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:36:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:36:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:36:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:36:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:36:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:36:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:36:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:36:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:36:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:36:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":4367467.73,"entries_behind_primary":1,"entries_per_second":272.17,"non_primary_position":{"entry_tid":1,"object_number":117,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":117,"tag_tid":19},"seconds_until_synced":0} 2024-05-08T00:36:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:36:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:36:56 2024-05-08T00:36:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:36:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:36:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:36:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:36:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:36:56' 2024-05-08T00:36:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:36:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:36:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:36:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":4367467.73,"entries_behind_primary":1,"entries_per_second":272.17,"non_primary_position":{"entry_tid":1,"object_number":117,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":117,"tag_tid":19},"seconds_until_synced":0} 2024-05-08T00:36:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:36:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:36:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:36:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:36:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:36:56.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:36:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:36:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:36:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:36:56' 2024-05-08T00:36:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:36:56' 2024-05-08T00:36:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:36:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:36:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:36:56' 2024-05-08T00:36:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:36:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:36:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:36:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":4367467.73,"entries_behind_primary":1,"entries_per_second":272.17,"non_primary_position":{"entry_tid":1,"object_number":117,"tag_tid":18},"primary_position":{"entry_tid":1,"object_number":117,"tag_tid":19},"seconds_until_synced":0}42594266mirror.0smithi1162024-05-08 00:36:56cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:36:56' 2024-05-08T00:36:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:36:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:36:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:36:56' 2024-05-08T00:36:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:36:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:36:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:36:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:36:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:36:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:36:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap7 2024-05-08T00:36:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap7 2024-05-08T00:36:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:36:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:36:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:36:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap7 2024-05-08T00:36:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:37:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:37:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:37:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:37:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:37:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:37:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:37:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:37:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:37:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:37:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:37:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:37:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:37:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:37:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:37:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:37:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:37:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:37:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:37:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:37:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:37:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:37:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:37:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:37:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:37:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:37:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:37:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:37:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:37:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:37:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:37:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:37:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:37:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:37:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:37:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:37:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:37:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:37:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:37:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:37:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:37:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:37:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:37:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:37:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:37:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:37:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:37:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:37:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 31 2024-05-08T00:37:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 31 2024-05-08T00:37:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:37:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=125, tag_tid=21, entry_tid=1], [object_number=124, tag_tid=21, entry_tid=0], [object_number=127, tag_tid=20, entry_tid=8103], [object_number=126, tag_tid=20, entry_tid=8102]]], state=connected] 2024-05-08T00:37:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=125, tag_tid=21, entry_tid=1], [object_number=124, tag_tid=21, entry_tid=0], [object_number=127, tag_tid=20, entry_tid=8103], [object_number=126, tag_tid=20, entry_tid=8102]]], state=connected] 2024-05-08T00:37:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:37:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=125, tag_tid=21, entry_tid=1' 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:37:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:37:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:37:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:37:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:37:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 31 2024-05-08T00:37:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 31 2024-05-08T00:37:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:37:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=125, tag_tid=21, entry_tid=1], [object_number=124, tag_tid=21, entry_tid=0], [object_number=127, tag_tid=20, entry_tid=8103], [object_number=126, tag_tid=20, entry_tid=8102]]], state=connected] 2024-05-08T00:37:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=125, tag_tid=21, entry_tid=1], [object_number=124, tag_tid=21, entry_tid=0], [object_number=127, tag_tid=20, entry_tid=8103], [object_number=126, tag_tid=20, entry_tid=8102]]], state=connected] 2024-05-08T00:37:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=125, tag_tid=21, entry_tid=1' 2024-05-08T00:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=125, tag_tid=21, entry_tid=1' -a 'object_number=125, tag_tid=21, entry_tid=1' = 'object_number=125, tag_tid=21, entry_tid=1' 2024-05-08T00:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-05-08T00:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap7 2024-05-08T00:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:37:27.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:37:27.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:37:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-05-08T00:37:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 13 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:35:49 2024 2024-05-08T00:37:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:36:33 2024 2024-05-08T00:37:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:37:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:37:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:37:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:37:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:37:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:37:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:37:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n snap6-clean ']' 2024-05-08T00:37:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap6-clean 2024-05-08T00:37:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap6-clean 2024-05-08T00:37:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:37:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:37:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:37:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:37:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:29.798 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:37:29.800+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:37:29.966 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:37:29.967+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:37:29.966 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:37:29.967+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:37:29.967 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:37:29.969+0000 7fbafa9ce640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-05-08T00:37:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:37:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:37:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap7 2024-05-08T00:37:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap7 2024-05-08T00:37:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:37:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:37:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:37:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:37:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:31.121 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:37:31.122+0000 7f2ed5201640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:37:31.224 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:37:31.225+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:37:31.225 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:37:31.225+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:37:31.293 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:37:31.294+0000 7fbafb1cf640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:37:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:37:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:37:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap7-clean 2024-05-08T00:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-05-08T00:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap7-clean 2024-05-08T00:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:37:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:37:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:37:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:37:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:37:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:37:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:37:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:37:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 52261888 bytes 2024-05-08T00:37:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 64707584 bytes 2024-05-08T00:37:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 80073728 bytes 2024-05-08T00:37:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 131581952 bytes 2024-05-08T00:37:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 187722752 bytes 2024-05-08T00:37:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:37:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:37:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:37:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:37:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:37:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:37:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:37:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:37:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:37:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":4329935.87,"entries_behind_primary":0,"entries_per_second":270.17,"non_primary_position":{"entry_tid":8097,"object_number":125,"tag_tid":20},"primary_position":{"entry_tid":4748,"object_number":120,"tag_tid":20}} 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:37:26 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:37:41' 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:37:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:37:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":4329935.87,"entries_behind_primary":0,"entries_per_second":270.17,"non_primary_position":{"entry_tid":8097,"object_number":125,"tag_tid":20},"primary_position":{"entry_tid":4748,"object_number":120,"tag_tid":20}} 2024-05-08T00:37:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:37:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:37:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:37:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:37:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:37:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:37:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:37:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:37:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:37:26' 2024-05-08T00:37:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:37:26' 2024-05-08T00:37:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:37:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:37:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:37:26' 2024-05-08T00:37:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:37:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:37:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:37:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":4329935.87,"entries_behind_primary":0,"entries_per_second":270.17,"non_primary_position":{"entry_tid":8097,"object_number":125,"tag_tid":20},"primary_position":{"entry_tid":4748,"object_number":120,"tag_tid":20}}42594266mirror.0smithi1162024-05-08 00:37:26cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:37:41' 2024-05-08T00:37:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:37:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:37:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:37:26' 2024-05-08T00:37:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:37:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:37:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:37:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:37:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:37:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:37:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:37:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap8 2024-05-08T00:37:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap8 2024-05-08T00:37:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:37:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:37:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:37:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap8 2024-05-08T00:37:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:37:50.730 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:37:50.730+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:38:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:38:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:38:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:38:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:38:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:38:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:38:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:38:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:38:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:38:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:38:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:38:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:38:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:38:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:38:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:38:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:38:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:38:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:38:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:38:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:38:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:38:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:38:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:38:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:38:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:38:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:38:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:38:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:38:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:38:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:38:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:38:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:38:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:38:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:38:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:38:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:38:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:38:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:38:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:38:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:38:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:38:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:38:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:38:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:38:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:38:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:38:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:38:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:38:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:38:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:38:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:38:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:38:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:38:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:38:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:38:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:38:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:38:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:38:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:38:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:38:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:38:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:38:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:38:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:38:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:38:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:38:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:38:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:38:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 35 2024-05-08T00:38:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 35 2024-05-08T00:38:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:38:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=141, tag_tid=24, entry_tid=1], [object_number=140, tag_tid=24, entry_tid=0], [object_number=143, tag_tid=23, entry_tid=11975], [object_number=142, tag_tid=23, entry_tid=11974]]], state=connected] 2024-05-08T00:38:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=141, tag_tid=24, entry_tid=1], [object_number=140, tag_tid=24, entry_tid=0], [object_number=143, tag_tid=23, entry_tid=11975], [object_number=142, tag_tid=23, entry_tid=11974]]], state=connected] 2024-05-08T00:38:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:38:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=141, tag_tid=24, entry_tid=1' 2024-05-08T00:38:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:38:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:38:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:38:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:38:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:38:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:38:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:38:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:38:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:38:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:38:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:38:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:38:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:38:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:38:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:38:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 35 2024-05-08T00:38:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 35 2024-05-08T00:38:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:38:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=141, tag_tid=24, entry_tid=1], [object_number=140, tag_tid=24, entry_tid=0], [object_number=143, tag_tid=23, entry_tid=11975], [object_number=142, tag_tid=23, entry_tid=11974]]], state=connected] 2024-05-08T00:38:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=141, tag_tid=24, entry_tid=1], [object_number=140, tag_tid=24, entry_tid=0], [object_number=143, tag_tid=23, entry_tid=11975], [object_number=142, tag_tid=23, entry_tid=11974]]], state=connected] 2024-05-08T00:38:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:38:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=141, tag_tid=24, entry_tid=1' 2024-05-08T00:38:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=141, tag_tid=24, entry_tid=1' -a 'object_number=141, tag_tid=24, entry_tid=1' = 'object_number=141, tag_tid=24, entry_tid=1' 2024-05-08T00:38:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:38:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-05-08T00:38:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:38:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap8 2024-05-08T00:38:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:38:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:38:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:38:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-05-08T00:38:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 15 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:37:29 2024 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:37:34 2024 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:38:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:38:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:38:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:38:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n snap7-clean ']' 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap7-clean 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap7-clean 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:38:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:38:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:16.759 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:38:16.760+0000 7f2ed41ff640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-05-08T00:38:16.760 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:38:16.761+0000 7f2ed7a06640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-05-08T00:38:16.832 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:38:16.833+0000 7f530a106640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 5 2024-05-08T00:38:16.977 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:38:16.978+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:38:16.978 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:38:16.979+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:38:17.163 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:38:17.162+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-05-08T00:38:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:38:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:38:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap8 2024-05-08T00:38:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap8 2024-05-08T00:38:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:38:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:38:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:38:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:38:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:18.422 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:38:18.422+0000 7f2eda20b640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-05-08T00:38:18.422 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:38:18.423+0000 7f2ed8207640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-05-08T00:38:18.635 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:38:18.636+0000 7f5307100640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-05-08T00:38:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:38:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:38:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:38:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap8-clean 2024-05-08T00:38:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-05-08T00:38:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:38:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap8-clean 2024-05-08T00:38:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:38:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:38:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:38:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:38:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:38:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=14 2024-05-08T00:38:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:38:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:38:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 59926528 bytes 2024-05-08T00:38:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 110092288 bytes 2024-05-08T00:38:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 168207360 bytes 2024-05-08T00:38:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 192846848 bytes 2024-05-08T00:38:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 211590144 bytes 2024-05-08T00:38:34.041 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:38:34.041+0000 7fa00a173640 -1 reset not still connected to 0x5564ecb41930 2024-05-08T00:38:34.041 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:38:34.041+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270410 2024-05-08T00:38:34.041 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:38:34.041+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271110 2024-05-08T00:38:34.041 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:38:34.041+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271790 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:38:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:38:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:38:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:38:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:38:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:38:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:38:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:38:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:38:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:38:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:38:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":3697686.5,"entries_behind_primary":1,"entries_per_second":230.43,"non_primary_position":{"entry_tid":1,"object_number":141,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":141,"tag_tid":25},"seconds_until_synced":0} 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:38:26 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:38:31' 2024-05-08T00:38:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:38:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:38:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:38:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":3697686.5,"entries_behind_primary":1,"entries_per_second":230.43,"non_primary_position":{"entry_tid":1,"object_number":141,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":141,"tag_tid":25},"seconds_until_synced":0} 2024-05-08T00:38:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:38:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:38:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:38:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:38:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:38:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:38:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:38:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:38:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:38:26' 2024-05-08T00:38:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:38:26' 2024-05-08T00:38:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:38:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:38:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:38:26' 2024-05-08T00:38:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:38:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:38:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:38:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":3697686.5,"entries_behind_primary":1,"entries_per_second":230.43,"non_primary_position":{"entry_tid":1,"object_number":141,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":141,"tag_tid":25},"seconds_until_synced":0}42594266mirror.0smithi1162024-05-08 00:38:26cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:38:31' 2024-05-08T00:38:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:38:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:38:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:38:26' 2024-05-08T00:38:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:38:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:38:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:38:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:38:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:38:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:38:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:38:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap9 2024-05-08T00:38:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap9 2024-05-08T00:38:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:38:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:38:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:38:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap9 2024-05-08T00:38:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:39:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:39:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:39:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:39:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:39:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:39:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:39:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:39:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:39:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:39:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:39:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:39:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:39:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:39:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:39:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:39:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:39:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:39:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:39:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:39:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:39:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:39:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:39:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:39:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:39:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:39:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:39:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:39:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:39:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:39:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:39:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:39:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:39:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:39:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:39:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:39:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:39:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:39:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:39:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=156, tag_tid=26, entry_tid=15584], [object_number=159, tag_tid=26, entry_tid=15583], [object_number=158, tag_tid=26, entry_tid=15582], [object_number=157, tag_tid=26, entry_tid=15581]]], state=connected] 2024-05-08T00:39:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:39:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=156, tag_tid=26, entry_tid=15584], [object_number=159, tag_tid=26, entry_tid=15583], [object_number=158, tag_tid=26, entry_tid=15582], [object_number=157, tag_tid=26, entry_tid=15581]]], state=connected] 2024-05-08T00:39:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=156, tag_tid=26, entry_tid=15584' 2024-05-08T00:39:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=161, tag_tid=27, entry_tid=1' -a 'object_number=161, tag_tid=27, entry_tid=1' = 'object_number=156, tag_tid=26, entry_tid=15584' 2024-05-08T00:39:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=156, tag_tid=26, entry_tid=15584' '!=' '' 2024-05-08T00:39:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:39:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=156, tag_tid=26, entry_tid=15584' = '' 2024-05-08T00:39:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=156, tag_tid=26, entry_tid=15584' 2024-05-08T00:39:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:39:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=27 2024-05-08T00:39:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=156, tag_tid=26, entry_tid=15584' 2024-05-08T00:39:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:39:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=26 2024-05-08T00:39:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:39:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:39:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=156, tag_tid=26, entry_tid=15584' 2024-05-08T00:39:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:39:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15584 2024-05-08T00:39:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 27 = 26 -a 1 -le 15584 2024-05-08T00:39:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:39:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:39:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:39:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:39:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:39:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:39:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:39:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:39:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:39:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:39:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:39:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:39:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:39:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:39:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:39:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:39:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=156, tag_tid=26, entry_tid=15792], [object_number=159, tag_tid=26, entry_tid=15791], [object_number=158, tag_tid=26, entry_tid=15790], [object_number=157, tag_tid=26, entry_tid=15789]]], state=connected] 2024-05-08T00:39:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:39:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=156, tag_tid=26, entry_tid=15792], [object_number=159, tag_tid=26, entry_tid=15791], [object_number=158, tag_tid=26, entry_tid=15790], [object_number=157, tag_tid=26, entry_tid=15789]]], state=connected] 2024-05-08T00:39:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=156, tag_tid=26, entry_tid=15792' 2024-05-08T00:39:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=161, tag_tid=27, entry_tid=1' -a 'object_number=161, tag_tid=27, entry_tid=1' = 'object_number=156, tag_tid=26, entry_tid=15792' 2024-05-08T00:39:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=156, tag_tid=26, entry_tid=15792' '!=' 'object_number=156, tag_tid=26, entry_tid=15584' 2024-05-08T00:39:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:39:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=156, tag_tid=26, entry_tid=15792' = 'object_number=156, tag_tid=26, entry_tid=15584' 2024-05-08T00:39:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=156, tag_tid=26, entry_tid=15792' 2024-05-08T00:39:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:39:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=27 2024-05-08T00:39:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=156, tag_tid=26, entry_tid=15792' 2024-05-08T00:39:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:39:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=26 2024-05-08T00:39:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:39:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:39:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=156, tag_tid=26, entry_tid=15792' 2024-05-08T00:39:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:39:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15792 2024-05-08T00:39:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 27 = 26 -a 1 -le 15792 2024-05-08T00:39:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:39:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:39:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:39:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:39:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:39:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:39:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:39:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:39:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846], [object_number=157, tag_tid=26, entry_tid=15845], [object_number=156, tag_tid=26, entry_tid=15844]]], state=connected] 2024-05-08T00:39:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:39:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846], [object_number=157, tag_tid=26, entry_tid=15845], [object_number=156, tag_tid=26, entry_tid=15844]]], state=connected] 2024-05-08T00:39:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=159, tag_tid=26, entry_tid=15847' 2024-05-08T00:39:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=161, tag_tid=27, entry_tid=1' -a 'object_number=161, tag_tid=27, entry_tid=1' = 'object_number=159, tag_tid=26, entry_tid=15847' 2024-05-08T00:39:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=159, tag_tid=26, entry_tid=15847' '!=' 'object_number=156, tag_tid=26, entry_tid=15792' 2024-05-08T00:39:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:39:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=159, tag_tid=26, entry_tid=15847' = 'object_number=156, tag_tid=26, entry_tid=15792' 2024-05-08T00:39:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=159, tag_tid=26, entry_tid=15847' 2024-05-08T00:39:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:39:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=27 2024-05-08T00:39:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=159, tag_tid=26, entry_tid=15847' 2024-05-08T00:39:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:39:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=26 2024-05-08T00:39:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:39:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:39:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=159, tag_tid=26, entry_tid=15847' 2024-05-08T00:39:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:39:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=15847 2024-05-08T00:39:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 27 = 26 -a 1 -le 15847 2024-05-08T00:39:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:39:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:39:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:39:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:39:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:39:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:39:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:39:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:39:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:39:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:39:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:39:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:39:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:39:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:39:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:39:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:39:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:39:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:39:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:39:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:39:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846], [object_number=157, tag_tid=26, entry_tid=15845], [object_number=156, tag_tid=26, entry_tid=15844]]], state=connected] 2024-05-08T00:39:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846], [object_number=157, tag_tid=26, entry_tid=15845], [object_number=156, tag_tid=26, entry_tid=15844]]], state=connected] 2024-05-08T00:39:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=159, tag_tid=26, entry_tid=15847' 2024-05-08T00:39:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=161, tag_tid=27, entry_tid=1' -a 'object_number=161, tag_tid=27, entry_tid=1' = 'object_number=159, tag_tid=26, entry_tid=15847' 2024-05-08T00:39:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=159, tag_tid=26, entry_tid=15847' '!=' 'object_number=159, tag_tid=26, entry_tid=15847' 2024-05-08T00:39:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:39:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.4 2024-05-08T00:39:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:39:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:39:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:39:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:39:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:39:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:39:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:39:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:39:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:39:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:39:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:39:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:39:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:39:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:39:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:39:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:39:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 39 2024-05-08T00:39:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 40 2024-05-08T00:39:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=161, tag_tid=27, entry_tid=1], [object_number=160, tag_tid=27, entry_tid=0], [object_number=159, tag_tid=26, entry_tid=15847], [object_number=158, tag_tid=26, entry_tid=15846]]], state=connected] 2024-05-08T00:39:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=161, tag_tid=27, entry_tid=1' -a 'object_number=161, tag_tid=27, entry_tid=1' = 'object_number=161, tag_tid=27, entry_tid=1' 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap9 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:39:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:39:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-05-08T00:39:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:39:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:39:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:39:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 17 2024-05-08T00:39:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:39:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:39:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:37:29 2024 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:38:35 2024 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:39:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n snap8-clean ']' 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap8-clean 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap8-clean 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:39:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:39:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:10.917 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:39:10.918+0000 7f2ed5201640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-05-08T00:39:10.997 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:39:10.998+0000 7f2eda20b640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:39:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:39:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap9 2024-05-08T00:39:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap9 2024-05-08T00:39:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:39:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:39:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:39:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:39:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:12.455 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:39:12.456+0000 7f2ed6203640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:39:12.525 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:39:12.526+0000 7f530a106640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-05-08T00:39:12.525 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:39:12.526+0000 7f5307100640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-05-08T00:39:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:39:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap9-clean 2024-05-08T00:39:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-05-08T00:39:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:39:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap9-clean 2024-05-08T00:39:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:39:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:39:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror test 2024-05-08T00:39:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:39:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:39:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=6 2024-05-08T00:39:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:39:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-08T00:39:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 74329088 bytes 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:39:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:39:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:39:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-08T00:39:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:39:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local description_pattern= 2024-05-08T00:39:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local service_pattern= 2024-05-08T00:39:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:39:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:39:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='test: 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: global_id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+replaying 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: replaying, {"bytes_per_second":5135531.1,"entries_behind_primary":1458,"entries_per_second":319.4,"non_primary_position":{"entry_tid":9407,"object_number":155,"tag_tid":26},"primary_position":{"entry_tid":10865,"object_number":153,"tag_tid":26},"seconds_until_synced":4} 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: service: mirror.0 on smithi116 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:38:56 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: peer_sites: 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: name: cluster2 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: state: up+stopped 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: description: local image is primary 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: last_update: 2024-05-08 00:39:16' 2024-05-08T00:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ state: .*up+replaying' 2024-05-08T00:39:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: state: up+replaying 2024-05-08T00:39:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^ description: .*' 2024-05-08T00:39:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: description: replaying, {"bytes_per_second":5135531.1,"entries_behind_primary":1458,"entries_per_second":319.4,"non_primary_position":{"entry_tid":9407,"object_number":155,"tag_tid":26},"primary_position":{"entry_tid":10865,"object_number":153,"tag_tid":26},"seconds_until_synced":4} 2024-05-08T00:39:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:39:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:39:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep '^up+' 2024-05-08T00:39:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:39:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep 'service: *mirror..* on ' 2024-05-08T00:39:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: service: mirror.0 on smithi116 2024-05-08T00:39:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update 2024-05-08T00:39:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-08T00:39:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:38:56' 2024-05-08T00:39:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-08 00:38:56' 2024-05-08T00:39:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state_pattern=up+replaying 2024-05-08T00:39:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'prev_last_update=2024-05-08 00:38:56' 2024-05-08T00:39:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status 2024-05-08T00:39:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ CEPH_ARGS= 2024-05-08T00:39:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-08T00:39:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status='OKOKOK142594266mirror.0smithi11619.0.0-2529-ge0c1691atrueOK42634269mirror.1smithi11619.0.0-2529-ge0c1691afalseOKtest9855df49-96e1-4286-bdc5-20cbb6a60ebcup+replayingreplaying, {"bytes_per_second":5135531.1,"entries_behind_primary":1458,"entries_per_second":319.4,"non_primary_position":{"entry_tid":9407,"object_number":155,"tag_tid":26},"primary_position":{"entry_tid":10865,"object_number":153,"tag_tid":26},"seconds_until_synced":4}42594266mirror.0smithi1162024-05-08 00:38:56cluster26516539d-24f8-4e22-af4f-42c9e9dd53dbup+stoppedlocal image is primary2024-05-08 00:39:16' 2024-05-08T00:39:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_update state 2024-05-08T00:39:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-08T00:39:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_update='2024-05-08 00:38:56' 2024-05-08T00:39:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-08T00:39:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=up+replaying 2024-05-08T00:39:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo up+replaying 2024-05-08T00:39:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep up+replaying 2024-05-08T00:39:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:up+replaying 2024-05-08T00:39:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap10 2024-05-08T00:39:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap10 2024-05-08T00:39:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:39:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap10 2024-05-08T00:39:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:39:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:39:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:39:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:39:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:39:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:39:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:39:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:39:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:39:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:39:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:39:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:39:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:39:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:39:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:39:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:39:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:39:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:39:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:39:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:39:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:39:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:39:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:39:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:39:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:39:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:39:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:39:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:39:52.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:39:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:39:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:39:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:39:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:39:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:39:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:39:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:39:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:39:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=29, entry_tid=7389], [object_number=168, tag_tid=29, entry_tid=7388], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:39:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:39:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:39:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:39:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=29, entry_tid=7389], [object_number=168, tag_tid=29, entry_tid=7388], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=169, tag_tid=30, entry_tid=1' -a 'object_number=169, tag_tid=30, entry_tid=1' = 'object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=169, tag_tid=29, entry_tid=7389' '!=' '' 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=169, tag_tid=29, entry_tid=7389' = '' 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=30 2024-05-08T00:39:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:39:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=29 2024-05-08T00:39:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:39:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:39:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:39:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:39:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:39:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=7389 2024-05-08T00:39:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 30 = 29 -a 1 -le 7389 2024-05-08T00:39:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:39:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:39:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:39:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:39:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:39:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:39:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:39:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:39:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:39:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:39:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:39:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:39:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:39:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:39:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:39:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=29, entry_tid=7389], [object_number=168, tag_tid=29, entry_tid=7388], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:39:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:39:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:39:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:39:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=29, entry_tid=7389], [object_number=168, tag_tid=29, entry_tid=7388], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:53.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=169, tag_tid=30, entry_tid=1' -a 'object_number=169, tag_tid=30, entry_tid=1' = 'object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:53.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=169, tag_tid=29, entry_tid=7389' '!=' 'object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:53.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:39:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.4 2024-05-08T00:39:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:39:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:39:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:39:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:39:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:39:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:39:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:39:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=29, entry_tid=7389], [object_number=168, tag_tid=29, entry_tid=7388], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:39:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:39:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:39:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:39:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:39:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:39:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=29, entry_tid=7389], [object_number=168, tag_tid=29, entry_tid=7388], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=169, tag_tid=30, entry_tid=1' -a 'object_number=169, tag_tid=30, entry_tid=1' = 'object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=169, tag_tid=29, entry_tid=7389' '!=' 'object_number=169, tag_tid=29, entry_tid=7389' 2024-05-08T00:39:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:39:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.8 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:39:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:39:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:39:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:39:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:39:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:39:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:39:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:39:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:39:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:39:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:39:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:39:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:39:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:39:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=169, tag_tid=30, entry_tid=1' -a 'object_number=169, tag_tid=30, entry_tid=1' = 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-05-08T00:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap10 2024-05-08T00:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:39:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:39:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:39:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-05-08T00:39:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:39:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:39:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:39:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 19 2024-05-08T00:39:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:39:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:39:10 2024 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:39:36 2024 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:39:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:39:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n snap9-clean ']' 2024-05-08T00:39:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap9-clean 2024-05-08T00:39:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap9-clean 2024-05-08T00:39:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:39:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:39:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:39:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:39:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:56.416 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:39:56.417+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:39:56.417 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:39:56.418+0000 7fbafa1cd640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 8 2024-05-08T00:39:56.818 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:39:56.819+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:39:57.034 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:39:57.035+0000 7fbafa9ce640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-05-08T00:39:57.035 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:39:57.036+0000 7fbaf99cc640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 10 2024-05-08T00:39:57.036 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:39:57.037+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 8 2024-05-08T00:39:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:39:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap10 2024-05-08T00:39:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap10 2024-05-08T00:39:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:39:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:39:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:39:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:39:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:58.049 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:39:58.050+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:39:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:39:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:39:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:39:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ clean_snap_name=snap10-clean 2024-05-08T00:39:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-05-08T00:39:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:39:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap10-clean 2024-05-08T00:39:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-05-08T00:39:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:39:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:39:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:40:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-08T00:40:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:40:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:40:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:40:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:40:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:40:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:40:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:40:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:40:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:40:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:40:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:40:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:40:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:40:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-08T00:40:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:40:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:40:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:40:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:40:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:40:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:40:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:40:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:40:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:40:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:40:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:40:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:40:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:40:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:40:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:40:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:40:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:40:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:40:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:40:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:40:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:40:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:40:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:40:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:40:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:40:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:40:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:40:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:40:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:40:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:40:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:40:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:40:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:40:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:40:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:40:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:40:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:40:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:40:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:40:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:40:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=169, tag_tid=31, entry_tid=1' 2024-05-08T00:40:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:40:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:40:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:40:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:40:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:40:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:40:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:40:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:40:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:40:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:40:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=169, tag_tid=31, entry_tid=1' -a 'object_number=169, tag_tid=31, entry_tid=1' = 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=169, tag_tid=30, entry_tid=1' '!=' '' 2024-05-08T00:40:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:40:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=169, tag_tid=30, entry_tid=1' = '' 2024-05-08T00:40:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=169, tag_tid=31, entry_tid=1' 2024-05-08T00:40:00.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:40:00.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:40:00.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=31 2024-05-08T00:40:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:40:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:40:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=30 2024-05-08T00:40:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=169, tag_tid=31, entry_tid=1' 2024-05-08T00:40:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:40:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:40:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:40:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:40:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:40:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=1 2024-05-08T00:40:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 31 = 30 -a 1 -le 1 2024-05-08T00:40:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:40:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:40:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:40:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:40:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:40:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:40:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:40:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:40:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:40:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:40:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:40:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:40:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:40:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:40:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:40:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:40:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:40:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:40:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:40:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:40:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:40:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:01.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=169, tag_tid=31, entry_tid=1' 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:40:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:40:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:40:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:40:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:40:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:40:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:40:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:40:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=169, tag_tid=31, entry_tid=1' -a 'object_number=169, tag_tid=31, entry_tid=1' = 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=169, tag_tid=30, entry_tid=1' '!=' 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:40:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.4 2024-05-08T00:40:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:40:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:40:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:40:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:40:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:40:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:40:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:40:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:40:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:40:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:40:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:40:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:40:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:40:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:40:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:40:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:40:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:40:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:40:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=169, tag_tid=31, entry_tid=1' 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:40:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:40:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:40:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:40:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:40:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:40:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:40:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:40:01.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:01.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=30, entry_tid=1], [object_number=168, tag_tid=30, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=169, tag_tid=31, entry_tid=1' -a 'object_number=169, tag_tid=31, entry_tid=1' = 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=169, tag_tid=30, entry_tid=1' '!=' 'object_number=169, tag_tid=30, entry_tid=1' 2024-05-08T00:40:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:40:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.8 2024-05-08T00:40:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror test 2024-05-08T00:40:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n test ']' 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/test' 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:40:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:40:02.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:40:02.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:40:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:40:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:40:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:40:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:40:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:40:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:40:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-05-08T00:40:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror test 2024-05-08T00:40:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '' 2024-05-08T00:40:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:40:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:40:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:40:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:40:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:40:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:40:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:40:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:40:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=169, tag_tid=31, entry_tid=1' 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-test.status 2024-05-08T00:40:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-test.status 2024-05-08T00:40:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:40:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-05-08T00:40:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 42 2024-05-08T00:40:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 42 2024-05-08T00:40:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:40:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=169, tag_tid=31, entry_tid=1], [object_number=168, tag_tid=31, entry_tid=0], [object_number=171, tag_tid=29, entry_tid=7387], [object_number=170, tag_tid=29, entry_tid=7386]]], state=connected] 2024-05-08T00:40:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.status 2024-05-08T00:40:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=169, tag_tid=31, entry_tid=1' 2024-05-08T00:40:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=169, tag_tid=31, entry_tid=1' -a 'object_number=169, tag_tid=31, entry_tid=1' = 'object_number=169, tag_tid=31, entry_tid=1' 2024-05-08T00:40:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-05-08T00:40:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap10-clean 2024-05-08T00:40:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:40:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:40:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:40:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'test': 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 20 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa640adee0 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa640adee0 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:31:28 2024 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:39:10 2024 2024-05-08T00:40:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:39:36 2024 2024-05-08T00:40:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:40:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa640adee0 2024-05-08T00:40:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:40:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:40:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 9855df49-96e1-4286-bdc5-20cbb6a60ebc 2024-05-08T00:40:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:40:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 10 2024-05-08T00:40:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap1 2024-05-08T00:40:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap1 2024-05-08T00:40:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap1 2024-05-08T00:40:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap1-clean 2024-05-08T00:40:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap1-clean 2024-05-08T00:40:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap1-clean 2024-05-08T00:40:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap2 2024-05-08T00:40:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap2 2024-05-08T00:40:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap2 2024-05-08T00:40:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:06.901 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:40:06.902+0000 7f5308102640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-05-08T00:40:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:07.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap2-clean 2024-05-08T00:40:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap2-clean 2024-05-08T00:40:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap2-clean 2024-05-08T00:40:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap3 2024-05-08T00:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap3 2024-05-08T00:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap3 2024-05-08T00:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:09.756 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:40:09.756+0000 7f53060fe640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-05-08T00:40:10.169 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:10.170+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:40:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap3-clean 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap3-clean 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap3-clean 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:11.210 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:11.211+0000 7f2ed5a02640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-05-08T00:40:11.211 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:11.212+0000 7f2ed8a08640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-05-08T00:40:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap4 2024-05-08T00:40:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap4 2024-05-08T00:40:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap4 2024-05-08T00:40:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap4-clean 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap4-clean 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap4-clean 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:14.277 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:14.277+0000 7f2ed7a06640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:40:14.278 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:14.277+0000 7f2ed4a00640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:40:14.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap5 2024-05-08T00:40:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap5 2024-05-08T00:40:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap5 2024-05-08T00:40:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:15.770 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:15.770+0000 7fbafa1cd640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-05-08T00:40:15.918 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:40:15.917+0000 7f53060fe640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-05-08T00:40:15.986 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:15.986+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:40:16.097 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:16.096+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:40:16.097 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:16.097+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 7 2024-05-08T00:40:16.283 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:16.283+0000 7fbaf99cc640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:40:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap5-clean 2024-05-08T00:40:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap5-clean 2024-05-08T00:40:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap5-clean 2024-05-08T00:40:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:17.519 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:40:17.519+0000 7f53060fe640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-05-08T00:40:17.911 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:17.911+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 7 2024-05-08T00:40:18.055 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:18.055+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:40:18.056 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:18.055+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:40:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap6 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap6 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap6 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:19.744 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:19.744+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:40:19.745 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:19.744+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:40:19.747 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:19.747+0000 7fbafc1d1640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 11 2024-05-08T00:40:19.822 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:19.822+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 5 2024-05-08T00:40:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap6-clean 2024-05-08T00:40:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap6-clean 2024-05-08T00:40:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap6-clean 2024-05-08T00:40:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:21.408 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:21.408+0000 7fbafa1cd640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:21.485 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:21.485+0000 7fbaf99cc640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:21.485 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:21.485+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:21.559 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:21.559+0000 7fbafc1d1640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-05-08T00:40:21.560 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:21.560+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-05-08T00:40:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap7 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap7 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap7 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:22.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:22.592+0000 7fbafc1d1640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:22.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:22.593+0000 7fbafb1cf640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:22.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:22.593+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:40:22.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:22.593+0000 7fbaf89ca640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:22.660 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:22.660+0000 7f2ed7a06640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-05-08T00:40:22.660 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:22.660+0000 7f2ed41ff640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-05-08T00:40:23.002 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:23.002+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-05-08T00:40:23.003 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:23.003+0000 7fbaf99cc640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 6 2024-05-08T00:40:23.200 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:23.200+0000 7fbafa9ce640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:23.202 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:23.202+0000 7fbafc1d1640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:40:23.202 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:23.202+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap7-clean 2024-05-08T00:40:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap7-clean 2024-05-08T00:40:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap7-clean 2024-05-08T00:40:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:24.214 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:24.214+0000 7fbafa1cd640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-05-08T00:40:24.323 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:24.323+0000 7f2ed9209640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:40:24.323 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:24.323+0000 7f2ed41ff640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:40:24.323 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:24.323+0000 7f2ed5a02640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:40:24.325 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:24.325+0000 7f2ed7a06640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:40:24.536 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:40:24.536+0000 7f53030f8640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-05-08T00:40:24.612 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:24.612+0000 7fbaf71c7640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:40:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:25.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap8 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap8 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap8 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:25.955 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:25.955+0000 7f2ed4a00640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:40:25.956 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:25.956+0000 7f2eda20b640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-05-08T00:40:26.316 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:26.313+0000 7fbaf99cc640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:40:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap8-clean 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap8-clean 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap8-clean 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:28.012 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:28.013+0000 7fbafa1cd640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:40:28.166 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:28.164+0000 7fbafa1cd640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-05-08T00:40:28.167 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:28.164+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-05-08T00:40:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap9 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap9 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap9 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:29.194 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:29.194+0000 7fbaf71c7640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:29.304 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:29.305+0000 7f2ed5201640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:40:29.304 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:29.305+0000 7f2ed41ff640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-05-08T00:40:29.305 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:29.306+0000 7f2ed7a06640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 3 2024-05-08T00:40:29.305 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:29.306+0000 7f2ed9a0a640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 5 2024-05-08T00:40:29.306 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:29.307+0000 7f2ed6203640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 1 2024-05-08T00:40:29.811 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:29.812+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 3 2024-05-08T00:40:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap9-clean 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap9-clean 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap9-clean 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:30.954 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:30.955+0000 7f2ed4a00640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-05-08T00:40:31.218 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:31.219+0000 7fbaf69c6640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 1 2024-05-08T00:40:31.361 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:31.362+0000 7fbaf59c4640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:40:31.508 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:31.508+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 4 2024-05-08T00:40:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap10 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap10 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap10 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:32.628 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:32.629+0000 7f2ed41ff640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 2 2024-05-08T00:40:32.696 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:40:32.697+0000 7f530a106640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-05-08T00:40:33.107 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:33.107+0000 7fbaf91cb640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 8 2024-05-08T00:40:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap10-clean 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror test snap10-clean 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap10-clean 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export 2024-05-08T00:40:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:34.312 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:40:34.313+0000 7f2ed5a02640 -1 bdev(0x56320c38ae00 /var/lib/ceph/osd/cluster1-1/block) aio_submit retries 4 2024-05-08T00:40:34.531 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:34.532+0000 7fbafd1d3640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 2 2024-05-08T00:40:34.533 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:40:34.533+0000 7fbaf61c5640 -1 bdev(0x55a77dfe0e00 /var/lib/ceph/osd/cluster1-2/block) aio_submit retries 7 2024-05-08T00:40:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-08T00:40:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-test.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-test.export 2024-05-08T00:40:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 10 2024-05-08T00:40:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap1 2024-05-08T00:40:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-05-08T00:40:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap1 2024-05-08T00:40:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-05-08T00:40:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap1-clean 2024-05-08T00:40:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-05-08T00:40:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap1-clean 2024-05-08T00:40:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-05-08T00:40:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap2 2024-05-08T00:40:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-05-08T00:40:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap2 2024-05-08T00:40:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-05-08T00:40:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap2-clean 2024-05-08T00:40:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-05-08T00:40:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap2-clean 2024-05-08T00:40:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-05-08T00:40:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap3 2024-05-08T00:40:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-05-08T00:40:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap3 2024-05-08T00:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-05-08T00:40:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap3-clean 2024-05-08T00:40:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-05-08T00:40:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap3-clean 2024-05-08T00:40:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-05-08T00:40:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap4 2024-05-08T00:40:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-05-08T00:40:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap4 2024-05-08T00:40:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-05-08T00:40:41.480 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:40:41.481+0000 7fa2f29c3640 -1 reset not still connected to 0x5640da7ad2b0 2024-05-08T00:40:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap4-clean 2024-05-08T00:40:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-05-08T00:40:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap4-clean 2024-05-08T00:40:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-05-08T00:40:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap5 2024-05-08T00:40:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-05-08T00:40:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap5 2024-05-08T00:40:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-05-08T00:40:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap5-clean 2024-05-08T00:40:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-05-08T00:40:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap5-clean 2024-05-08T00:40:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-05-08T00:40:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap6 2024-05-08T00:40:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-05-08T00:40:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap6 2024-05-08T00:40:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-05-08T00:40:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap6-clean 2024-05-08T00:40:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-05-08T00:40:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap6-clean 2024-05-08T00:40:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-05-08T00:40:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap7 2024-05-08T00:40:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-05-08T00:40:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap7 2024-05-08T00:40:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-05-08T00:40:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap7-clean 2024-05-08T00:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-05-08T00:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap7-clean 2024-05-08T00:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-05-08T00:40:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap8 2024-05-08T00:40:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-05-08T00:40:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap8 2024-05-08T00:40:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-05-08T00:40:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap8-clean 2024-05-08T00:40:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-05-08T00:40:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap8-clean 2024-05-08T00:40:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-05-08T00:40:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap9 2024-05-08T00:40:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-05-08T00:40:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap9 2024-05-08T00:40:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-05-08T00:40:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap9-clean 2024-05-08T00:40:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-05-08T00:40:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap9-clean 2024-05-08T00:40:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-05-08T00:40:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:40:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap10 2024-05-08T00:40:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-05-08T00:40:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap10 2024-05-08T00:40:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-05-08T00:40:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap10-clean 2024-05-08T00:40:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-05-08T00:40:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap10-clean 2024-05-08T00:40:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-05-08T00:40:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:40:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror test 2024-05-08T00:40:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:40:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:40:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror test 2024-05-08T00:40:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-05-08T00:40:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-05-08T00:40:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T00:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-05-08T00:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:40:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:40:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror test 2024-05-08T00:40:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:40:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-05-08T00:40:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:40:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aa640adee0 2024-05-08T00:40:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:40:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror test deleted 10aa640adee0 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aa640adee0 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror test 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-05-08T00:40:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:40:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10aa640adee0 2024-05-08T00:40:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10aa640adee0 2024-05-08T00:40:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10aa640adee0 -o 10aa640adee0 = 10aa640adee0 2024-05-08T00:40:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:40:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = present 2024-05-08T00:40:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:40:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror test deleted 10aa640adee0 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=test 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aa640adee0 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror test 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:40:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=test 2024-05-08T00:40:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-05-08T00:40:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:40:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image test: (2) No such file or directory 2024-05-08T00:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ testlog 'TEST: create many images' 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ tee -a /tmp/tmp.W1VfRby4SU/rbd-mirror.test.log 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ date '+%F %T' 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo 2024-05-08 00:40:57 TEST: create many images 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:2024-05-08 00:40:57 TEST: create many images 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ snap_name=snap 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 50 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_1 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 journal 128M 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_1 128M 2024-05-08T00:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:40:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-05-08T00:40:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_1 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:40:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:40:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:40:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:40:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:41:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 40601600 bytes 2024-05-08T00:41:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 69612544 bytes 2024-05-08T00:41:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 141310976 bytes 2024-05-08T00:41:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 184330240 bytes 2024-05-08T00:41:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:41:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:41:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:41:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:41:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:41:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_2 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 journal 128M 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_2 128M 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-05-08T00:41:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_2 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:41:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:41:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=8 2024-05-08T00:41:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:41:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:41:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 5266432 bytes 2024-05-08T00:41:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 40173568 bytes 2024-05-08T00:41:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 52102144 bytes 2024-05-08T00:41:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 78986240 bytes 2024-05-08T00:41:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 89310208 bytes 2024-05-08T00:41:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 135280640 bytes 2024-05-08T00:41:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:700 IOs, 163832832 bytes 2024-05-08T00:41:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_3 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 journal 128M 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:41:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_3 128M 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-05-08T00:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:41:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:41:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:41:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:41:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_3 2024-05-08T00:41:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:41:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:41:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=14 2024-05-08T00:41:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:41:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:41:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 40324096 bytes 2024-05-08T00:41:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 53928960 bytes 2024-05-08T00:41:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 139888640 bytes 2024-05-08T00:41:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 200270848 bytes 2024-05-08T00:41:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 242230272 bytes 2024-05-08T00:41:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 260959232 bytes 2024-05-08T00:41:29.430 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.430+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f4a04e0 2024-05-08T00:41:29.430 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.430+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f7c0000 2024-05-08T00:41:29.430 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.430+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:41:29.430 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.430+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:41:29.430 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.430+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:41:29.709 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.709+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:41:29.709 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.709+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:41:29.709 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.709+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:41:29.710 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.711+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:41:29.711 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.712+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:41:29.712 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.712+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:41:29.897 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.898+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f7c0000 2024-05-08T00:41:29.898 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.899+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:41:29.898 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.899+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:41:29.898 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:29.899+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:41:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:41:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:41:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_4 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 journal 128M 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_4 128M 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-05-08T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_4 2024-05-08T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:41:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:41:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=10 2024-05-08T00:41:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:41:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:41:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 30696448 bytes 2024-05-08T00:41:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 171872256 bytes 2024-05-08T00:41:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_5 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 journal 128M 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_5 128M 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:41:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:41:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:41:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:41:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_5 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:41:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:41:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=8 2024-05-08T00:41:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:41:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:41:41.167 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:41:41.167+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b8d00 2024-05-08T00:41:41.167 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:41:41.167+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:41:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 41093120 bytes 2024-05-08T00:41:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 84830208 bytes 2024-05-08T00:41:45.868 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:41:45.869+0000 7fa2f29c3640 -1 reset not still connected to 0x5640e260a680 2024-05-08T00:41:45.871 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:41:45.870+0000 7fa2f29c3640 -1 reset not still connected to 0x5640e260be10 2024-05-08T00:41:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:41:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:41:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:41:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:41:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:41:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_6 2024-05-08T00:41:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 journal 128M 2024-05-08T00:41:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_6 128M 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:41:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:41:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:41:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:41:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-05-08T00:41:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:41:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:41:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:41:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:41:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:41:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_6 2024-05-08T00:41:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:41:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:41:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=10 2024-05-08T00:41:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:41:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:41:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 57126912 bytes 2024-05-08T00:41:54.341 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.339+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:41:54.341 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.339+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:41:54.341 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.339+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:41:54.341 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.341+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:41:54.736 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.737+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:41:54.736 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.737+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:41:54.738 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.738+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:41:54.738 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.738+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:41:54.738 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.738+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:41:54.738 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:54.738+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:41:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 109924352 bytes 2024-05-08T00:41:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 158664704 bytes 2024-05-08T00:41:56.655 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:56.655+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f7c0000 2024-05-08T00:41:56.661 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:56.658+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:41:56.661 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:56.658+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:41:56.661 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:56.658+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:41:56.662 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:41:56.658+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:41:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:41:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:41:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:41:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:41:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_7 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 journal 128M 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_7 128M 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:41:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-05-08T00:41:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:41:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:41:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:41:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:41:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:41:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_7 2024-05-08T00:41:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:41:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:41:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:41:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:41:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=10 2024-05-08T00:41:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:41:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:42:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 48096256 bytes 2024-05-08T00:42:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 69953536 bytes 2024-05-08T00:42:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 86169600 bytes 2024-05-08T00:42:04.497 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:04.497+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:42:04.497 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:04.497+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:42:04.498 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:04.497+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:42:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 122952704 bytes 2024-05-08T00:42:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 146563072 bytes 2024-05-08T00:42:06.821 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:06.821+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0ba00 2024-05-08T00:42:06.823 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:06.823+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:42:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 173620224 bytes 2024-05-08T00:42:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_8 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 journal 128M 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_8 128M 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_8 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:42:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:42:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:42:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:42:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:42:09.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 19428352 bytes 2024-05-08T00:42:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 97065984 bytes 2024-05-08T00:42:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 115126272 bytes 2024-05-08T00:42:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 156876800 bytes 2024-05-08T00:42:17.503 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:17.503+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:42:17.505 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:17.504+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:42:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 211298304 bytes 2024-05-08T00:42:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_9 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 journal 128M 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_9 128M 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:42:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:42:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-05-08T00:42:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:42:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:42:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:42:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:42:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:42:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_9 2024-05-08T00:42:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:42:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:42:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=7 2024-05-08T00:42:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:42:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 35600384 bytes 2024-05-08T00:42:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 87955456 bytes 2024-05-08T00:42:24.309 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:24.310+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:42:24.315 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:24.315+0000 7fbb0a622640 -1 reset not still connected to 0x55a78c115ad0 2024-05-08T00:42:24.315 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:24.315+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:42:25.218 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:25.219+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:42:25.218 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:25.219+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:42:26.591 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:26.591+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:42:26.591 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:26.591+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:42:26.591 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:26.591+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:42:26.592 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:26.593+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:42:26.594 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:26.595+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:42:26.594 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:26.595+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:42:26.594 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:26.595+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:42:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_10 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 journal 128M 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:42:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_10 128M 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:42:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-05-08T00:42:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:42:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:42:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:42:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:42:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:42:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_10 2024-05-08T00:42:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:42:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:42:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:42:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:42:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:42:29.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 39810048 bytes 2024-05-08T00:42:29.885 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:29.885+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:42:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 74052608 bytes 2024-05-08T00:42:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 74653696 bytes 2024-05-08T00:42:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 137970688 bytes 2024-05-08T00:42:35.165 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:35.165+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:42:35.165 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:35.165+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:42:35.630 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:35.630+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cca90 2024-05-08T00:42:35.631 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:35.630+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:42:35.631 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:42:35.632+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_11 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 journal 128M 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_11 128M 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:42:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:42:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:42:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_11 2024-05-08T00:42:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:42:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:42:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:42:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:42:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:42:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_11 2024-05-08T00:42:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:42:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:42:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=6 2024-05-08T00:42:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:42:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:42:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 13030400 bytes 2024-05-08T00:42:38.493 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.493+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:42:38.493 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.493+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:42:38.493 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.493+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:42:38.493 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.493+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:42:38.493 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.493+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:42:38.494 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.493+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:42:38.572 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.571+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:42:38.874 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.874+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:42:38.874 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.874+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:42:38.874 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.874+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:42:38.875 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.876+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:42:38.875 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.876+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:42:38.876 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.876+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:42:38.876 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.876+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:42:38.876 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.876+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:42:38.876 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.876+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:42:38.876 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:38.876+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:42:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 43611136 bytes 2024-05-08T00:42:40.437 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:40.438+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b9ee0 2024-05-08T00:42:40.438 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:40.438+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:42:40.438 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:40.438+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:42:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 82144256 bytes 2024-05-08T00:42:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:42:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:42:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:42:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:42:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:42:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_12 2024-05-08T00:42:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 journal 128M 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_12 128M 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:42:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:42:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:42:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:42:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_12 2024-05-08T00:42:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:42:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_12 2024-05-08T00:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:42:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:42:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=10 2024-05-08T00:42:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:42:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:42:42.602 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:42.600+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:42:42.602 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:42.600+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:42:42.602 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:42.601+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:42:42.603 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:42.601+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:42:42.603 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:42.602+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:42:42.603 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:42.602+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:42:43.161 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:43.162+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:42:43.163 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:43.163+0000 7fbb0a622640 -1 reset not still connected to 0x55a78c115ad0 2024-05-08T00:42:43.163 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:43.163+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:42:43.389 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:43.389+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:42:43.391 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:43.391+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:42:43.391 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:43.391+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:42:43.391 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:43.391+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:42:43.392 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:43.391+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:42:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 48535552 bytes 2024-05-08T00:42:45.444 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:45.443+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:42:45.445 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:45.445+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:42:45.450 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:45.451+0000 7fbb0a622640 -1 reset not still connected to 0x55a78c115ad0 2024-05-08T00:42:45.452 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:45.453+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:42:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 57623552 bytes 2024-05-08T00:42:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 61329408 bytes 2024-05-08T00:42:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 72315904 bytes 2024-05-08T00:42:48.072 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:48.071+0000 7f5317d5e640 -1 reset not still connected to 0x562084709450 2024-05-08T00:42:48.072 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:48.071+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:42:48.073 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:48.071+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:42:48.073 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:48.071+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:42:48.073 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:48.071+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:42:48.073 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:48.071+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:42:48.127 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:48.126+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:42:48.127 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:48.126+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:42:48.127 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:48.126+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:42:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 93907968 bytes 2024-05-08T00:42:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 94527488 bytes 2024-05-08T00:42:49.213 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:49.213+0000 7f5317d5e640 -1 reset not still connected to 0x562084709450 2024-05-08T00:42:49.213 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:49.213+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:42:49.214 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:49.213+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:42:49.214 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:49.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:42:49.214 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:49.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:42:49.214 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:42:49.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:42:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:700 IOs, 143566848 bytes 2024-05-08T00:42:51.777 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:42:51.777+0000 7fa2f29c3640 -1 reset not still connected to 0x5640d3e916c0 2024-05-08T00:42:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_13 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 journal 128M 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_13 128M 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:42:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:42:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_13 2024-05-08T00:42:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:42:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:42:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:42:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:42:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:42:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_13 2024-05-08T00:42:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:42:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:42:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:42:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:42:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:42:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:42:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:42:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 24861696 bytes 2024-05-08T00:42:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 47991808 bytes 2024-05-08T00:42:57.168 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:57.167+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:42:57.168 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:57.167+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:42:57.170 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:42:57.170+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:42:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 86690816 bytes 2024-05-08T00:43:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 128442368 bytes 2024-05-08T00:43:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 178430976 bytes 2024-05-08T00:43:02.872 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:02.872+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:43:02.872 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:02.872+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:43:02.872 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:02.873+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:43:02.873 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:02.873+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:43:02.911 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:02.911+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:43:02.911 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:02.912+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:43:02.912 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:02.912+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:43:02.912 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:02.912+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_14 2024-05-08T00:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 journal 128M 2024-05-08T00:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:03.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_14 128M 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:43:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:43:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:43:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_14 2024-05-08T00:43:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:43:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:43:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:43:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:43:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:43:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_14 2024-05-08T00:43:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:43:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:43:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:43:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:43:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:43:03.434 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:03.433+0000 7f5317d5e640 -1 reset not still connected to 0x562084709450 2024-05-08T00:43:03.434 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:03.433+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:43:03.436 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:03.437+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:43:03.439 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:03.439+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:43:03.439 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:03.439+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:43:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 18451456 bytes 2024-05-08T00:43:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 49305600 bytes 2024-05-08T00:43:08.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 102368256 bytes 2024-05-08T00:43:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 125062144 bytes 2024-05-08T00:43:12.556 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:12.556+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:43:12.558 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:12.559+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:43:12.560 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:12.559+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:43:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 203776000 bytes 2024-05-08T00:43:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_15 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 journal 128M 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_15 128M 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:43:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:43:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_15 2024-05-08T00:43:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:43:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:43:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:43:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:43:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:43:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_15 2024-05-08T00:43:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:43:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:43:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=6 2024-05-08T00:43:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:43:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:43:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 30139392 bytes 2024-05-08T00:43:15.427 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:15.427+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:43:15.427 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:15.427+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:43:15.427 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:15.427+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:43:15.431 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:15.431+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:43:15.431 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:15.431+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:43:15.431 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:15.431+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:43:15.431 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:15.431+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 84614144 bytes 2024-05-08T00:43:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:43:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:43:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:43:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:43:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:43:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_16 2024-05-08T00:43:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 journal 128M 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_16 128M 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:43:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:43:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:43:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:43:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_16 2024-05-08T00:43:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:43:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:43:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:43:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:43:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:43:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_16 2024-05-08T00:43:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:43:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:43:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:43:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:43:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:43:25.705 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:25.705+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:43:25.705 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:25.705+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:43:25.706 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:25.706+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:43:25.706 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:25.706+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:43:25.706 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:25.706+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:43:25.706 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:25.706+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:43:26.384 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:26.382+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f7c0000 2024-05-08T00:43:26.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:26.384+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:43:26.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:26.384+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:43:26.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:26.384+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:43:26.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:26.385+0000 7fbb0a622640 -1 reset not still connected to 0x55a781903110 2024-05-08T00:43:26.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:26.385+0000 7fbb0a622640 -1 reset not still connected to 0x55a78c115ad0 2024-05-08T00:43:26.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:26.385+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:43:26.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:26.385+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:43:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 88907776 bytes 2024-05-08T00:43:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 138065920 bytes 2024-05-08T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_17 2024-05-08T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 journal 128M 2024-05-08T00:43:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:43:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_17 128M 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:43:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_17 2024-05-08T00:43:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:43:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:43:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:43:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:43:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:43:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_17 2024-05-08T00:43:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:43:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:43:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=10 2024-05-08T00:43:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:43:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:43:33.428 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:33.429+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:43:33.429 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:33.429+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:43:33.429 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:33.429+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:43:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 36459520 bytes 2024-05-08T00:43:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 49269760 bytes 2024-05-08T00:43:35.763 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:35.763+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:43:35.763 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:35.763+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:43:37.079 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:37.078+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:43:37.079 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:37.078+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:43:37.079 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:37.078+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:43:37.145 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:37.143+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:43:37.146 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:37.145+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:43:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 89984000 bytes 2024-05-08T00:43:37.914 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:37.913+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:43:37.914 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:37.913+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:43:37.914 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:37.913+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:43:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 101129216 bytes 2024-05-08T00:43:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 136005632 bytes 2024-05-08T00:43:41.233 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:41.232+0000 7f2ee8664640 -1 reset not still connected to 0x5632159288f0 2024-05-08T00:43:41.234 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:41.234+0000 7f2ee8664640 -1 reset not still connected to 0x56321c360c30 2024-05-08T00:43:41.239 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:41.239+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:43:41.240 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:41.239+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:43:41.240 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:41.239+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:43:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:43:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_18 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 journal 128M 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:43:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_18 128M 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_18 2024-05-08T00:43:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:43:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:43:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:43:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:43:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:43:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_18 2024-05-08T00:43:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:43:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:43:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=14 2024-05-08T00:43:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:43:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:43:42.308 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:42.305+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:43:42.308 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:42.305+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:43:43.669 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:43.668+0000 7f5317d5e640 -1 reset not still connected to 0x5620830b40d0 2024-05-08T00:43:43.674 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:43.673+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b8d00 2024-05-08T00:43:43.675 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:43.675+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:43:43.675 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:43.675+0000 7f5317d5e640 -1 reset not still connected to 0x5620830b40d0 2024-05-08T00:43:43.675 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:43.675+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b8d00 2024-05-08T00:43:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:43.675+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:43:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:43.675+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:43:43.676 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:43.675+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:43:43.831 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:43.831+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:43:43.832 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:43.831+0000 7f2ee8664640 -1 reset not still connected to 0x5632159288f0 2024-05-08T00:43:43.832 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:43.831+0000 7f2ee8664640 -1 reset not still connected to 0x56321c360c30 2024-05-08T00:43:43.832 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:43.831+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:43:43.834 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:43.834+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:43:43.834 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:43.834+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:43:44.194 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:44.193+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:43:44.194 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:44.193+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:43:44.194 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:44.193+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:43:44.798 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:44.798+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:43:47.346 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:43:47.347+0000 7fa2e3d69640 -1 bdev(0x5640ce37ee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-05-08T00:43:47.348 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:43:47.349+0000 7fa2dfd61640 -1 bdev(0x5640ce37ee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 7 2024-05-08T00:43:47.352 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:43:47.353+0000 7fa2e2d67640 -1 bdev(0x5640ce37ee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 1 2024-05-08T00:43:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 64010240 bytes 2024-05-08T00:43:48.072 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:48.071+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:43:48.073 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:48.073+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:43:48.073 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:48.073+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:43:48.073 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:48.073+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:43:48.902 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:48.902+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f7c0000 2024-05-08T00:43:48.906 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:48.906+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:43:48.906 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:48.906+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:43:48.911 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:48.911+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:43:48.912 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:48.911+0000 7fbb0a622640 -1 reset not still connected to 0x55a781903110 2024-05-08T00:43:48.912 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:48.911+0000 7fbb0a622640 -1 reset not still connected to 0x55a78c115ad0 2024-05-08T00:43:48.912 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:48.911+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:43:48.912 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:48.911+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:43:48.912 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:48.911+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:43:49.851 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:49.849+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:43:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 101874688 bytes 2024-05-08T00:43:52.158 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:52.158+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b8d00 2024-05-08T00:43:52.160 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:52.161+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b9ee0 2024-05-08T00:43:52.160 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:52.161+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:43:52.160 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:52.161+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:43:52.161 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:52.161+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:43:52.161 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:52.161+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:43:52.161 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:52.161+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:43:52.161 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:52.161+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:43:52.675 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:52.675+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:43:52.678 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:52.677+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:43:53.144 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:53.144+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:43:53.526 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:53.526+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:43:54.505 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:54.506+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:43:54.506 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:54.507+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:43:54.509 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:54.510+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:43:54.509 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:54.510+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:43:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 167835648 bytes 2024-05-08T00:43:54.957 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:54.956+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:43:54.957 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:54.956+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:43:54.957 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:54.956+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:43:55.234 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:55.231+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:43:55.234 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:55.231+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:43:55.234 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:55.231+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:43:55.234 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:55.231+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:43:55.234 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:55.231+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:43:55.234 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:43:55.231+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:43:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_19 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 journal 128M 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:43:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_19 128M 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:43:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:43:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:43:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_19 2024-05-08T00:43:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_19 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:43:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:43:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:43:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:43:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:43:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 42554368 bytes 2024-05-08T00:43:57.286 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:57.286+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:43:57.291 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:57.288+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:43:57.291 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:57.288+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:43:57.291 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:57.288+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:43:59.164 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:59.164+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0ba00 2024-05-08T00:43:59.166 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:43:59.166+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:43:59.212 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:59.213+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:43:59.212 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:59.213+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:43:59.218 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:59.218+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:43:59.218 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:59.218+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:43:59.218 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:59.218+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:43:59.218 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:59.218+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:43:59.218 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:59.218+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:43:59.218 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:43:59.218+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:44:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 83756032 bytes 2024-05-08T00:44:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:44:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_20 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 journal 128M 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_20 128M 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_20 2024-05-08T00:44:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:44:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:44:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:44:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:44:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:44:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_20 2024-05-08T00:44:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:44:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:44:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:44:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:44:06.123 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.119+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b8d00 2024-05-08T00:44:06.124 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.119+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b9ee0 2024-05-08T00:44:06.124 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.119+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:44:06.124 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.123+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:44:06.125 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.126+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:44:06.125 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.126+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:44:06.125 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.126+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:44:06.125 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.126+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:44:06.338 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.337+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:44:06.339 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:06.338+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:44:07.347 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:07.343+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:44:07.347 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:07.343+0000 7f2ee8664640 -1 reset not still connected to 0x5632159288f0 2024-05-08T00:44:07.348 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:07.343+0000 7f2ee8664640 -1 reset not still connected to 0x56321c360c30 2024-05-08T00:44:07.348 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:07.343+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:44:07.348 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:07.343+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:44:07.348 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:07.343+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:44:07.348 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:07.343+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:44:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 57065472 bytes 2024-05-08T00:44:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 81753088 bytes 2024-05-08T00:44:10.021 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:44:10.020+0000 7fa00a173640 -1 reset not still connected to 0x5564f16fd790 2024-05-08T00:44:10.021 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:44:10.020+0000 7fa00a173640 -1 reset not still connected to 0x5564f19b6dd0 2024-05-08T00:44:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 101677056 bytes 2024-05-08T00:44:13.197 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:13.196+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:44:13.203 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:13.203+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:44:13.210 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:13.211+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:44:13.210 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:13.211+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:44:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 136030208 bytes 2024-05-08T00:44:15.369 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.369+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:44:15.373 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.374+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:44:15.957 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.958+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0a9c0 2024-05-08T00:44:15.957 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.958+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0ba00 2024-05-08T00:44:15.957 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.958+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:44:15.957 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.958+0000 7f2ee8664640 -1 reset not still connected to 0x5632159288f0 2024-05-08T00:44:15.958 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.958+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:44:15.958 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.958+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:44:15.958 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.958+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:44:15.958 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.958+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:44:15.977 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.978+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0a9c0 2024-05-08T00:44:15.978 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.978+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0ba00 2024-05-08T00:44:15.978 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.978+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:44:15.978 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.978+0000 7f2ee8664640 -1 reset not still connected to 0x5632159288f0 2024-05-08T00:44:15.978 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.978+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:44:15.978 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.978+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:44:15.978 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.978+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:44:15.979 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:15.980+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:44:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:44:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:44:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:44:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:44:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:44:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_21 2024-05-08T00:44:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 journal 128M 2024-05-08T00:44:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:44:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:44:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_21 128M 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:44:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:44:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_21 2024-05-08T00:44:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:44:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:44:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:44:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:44:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:44:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_21 2024-05-08T00:44:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:44:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:44:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:44:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:44:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:44:16.662 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:16.663+0000 7f2ee8664640 -1 reset not still connected to 0x563219425040 2024-05-08T00:44:16.662 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:16.663+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:44:16.662 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:16.663+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:44:16.663 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:16.663+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:44:16.663 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:16.663+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:44:16.749 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:16.748+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b8d00 2024-05-08T00:44:16.749 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:16.748+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:44:16.749 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:16.748+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:44:16.749 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:16.748+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:44:16.749 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:16.748+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:44:16.749 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:16.748+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:44:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 9097216 bytes 2024-05-08T00:44:17.109 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:17.109+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:44:17.112 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:17.113+0000 7f2ee8664640 -1 reset not still connected to 0x5632159288f0 2024-05-08T00:44:17.112 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:17.113+0000 7f2ee8664640 -1 reset not still connected to 0x56321c360c30 2024-05-08T00:44:17.112 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:17.113+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:44:17.113 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:17.113+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:44:17.113 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:17.113+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:44:17.113 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:17.113+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:44:17.882 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:17.882+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:44:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 43382784 bytes 2024-05-08T00:44:20.888 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:20.889+0000 7fbb0a622640 -1 reset not still connected to 0x55a78c115ad0 2024-05-08T00:44:20.890 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:20.891+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:44:20.891 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:20.891+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:44:20.891 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:20.891+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:44:21.094 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.094+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:44:21.094 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.094+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:44:21.099 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.100+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:44:21.099 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.100+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:44:21.099 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.100+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:44:21.099 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.100+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:44:21.099 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.100+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:44:21.343 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.343+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:44:21.343 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.343+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:44:21.343 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.343+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:44:21.344 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.345+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:44:21.345 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.345+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:44:21.345 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.345+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:44:21.345 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.345+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:44:21.365 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.366+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:44:21.365 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.366+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:44:21.366 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.366+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:44:21.366 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.367+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:44:21.368 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.368+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:44:21.368 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.368+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:44:21.368 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.368+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:44:21.368 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.368+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:44:21.369 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.368+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:44:21.369 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.368+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:44:21.369 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.368+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:44:21.369 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.368+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:44:21.369 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:21.368+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:44:21.648 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.648+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b8d00 2024-05-08T00:44:21.652 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.653+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b9ee0 2024-05-08T00:44:21.653 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.653+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:44:21.653 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.653+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:44:21.653 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.653+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:44:21.653 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.653+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:44:21.653 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.653+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:44:21.653 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.653+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:44:21.653 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:21.653+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:44:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 71844864 bytes 2024-05-08T00:44:23.184 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:23.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:44:23.186 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:23.186+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:44:23.186 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:23.186+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:44:23.186 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:23.186+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:44:23.186 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:23.186+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:44:23.186 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:23.186+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:44:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 115081216 bytes 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_22 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 journal 128M 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_22 128M 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:44:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_22 2024-05-08T00:44:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:44:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:44:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:44:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:44:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:44:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_22 2024-05-08T00:44:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:44:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:44:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:44:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:44:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:44:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 10263552 bytes 2024-05-08T00:44:26.666 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:26.666+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:44:26.666 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:26.666+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:44:26.666 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:26.666+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:44:26.667 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:26.667+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:44:27.581 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:27.582+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:44:29.887 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:29.886+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:44:29.887 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:29.887+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:44:29.889 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:29.889+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:44:29.891 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:29.890+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:44:29.891 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:29.890+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:44:29.891 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:29.890+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:44:29.891 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:29.890+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:44:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 74543104 bytes 2024-05-08T00:44:31.558 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:31.557+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:44:31.558 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:31.557+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:44:31.558 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:31.558+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:44:31.562 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:31.561+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:44:31.562 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:31.561+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:44:31.562 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:31.561+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:44:31.562 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:31.561+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:44:31.562 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:31.561+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:44:31.562 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:31.561+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:44:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 122708992 bytes 2024-05-08T00:44:32.621 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:32.620+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:44:32.621 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:32.620+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:44:32.621 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:32.620+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:44:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_23 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 journal 128M 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_23 128M 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:44:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:44:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_23 2024-05-08T00:44:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_23 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:44:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:44:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=8 2024-05-08T00:44:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:44:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:44:36.545 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.536+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:44:36.545 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.536+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:44:36.548 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.542+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:44:36.549 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.545+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:44:36.550 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.548+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:44:36.550 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.548+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:44:36.550 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.548+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:44:36.550 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.548+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:44:36.551 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.550+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:44:36.551 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:36.550+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:44:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 3566592 bytes 2024-05-08T00:44:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 13673472 bytes 2024-05-08T00:44:37.720 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:37.718+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:44:37.720 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:37.718+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:44:37.721 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:37.720+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:44:37.721 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:37.720+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:44:37.721 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:44:37.720+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:44:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 31125504 bytes 2024-05-08T00:44:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 47722496 bytes 2024-05-08T00:44:40.693 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:44:40.691+0000 7fa2f29c3640 -1 reset not still connected to 0x5640eeb55380 2024-05-08T00:44:40.693 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:44:40.692+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f23055f0 2024-05-08T00:44:40.774 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:44:40.771+0000 7fa00a173640 -1 reset not still connected to 0x5564ee15fad0 2024-05-08T00:44:40.775 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:44:40.771+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270270 2024-05-08T00:44:40.784 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:44:40.777+0000 7fa00a173640 -1 reset not still connected to 0x5564fd97ca90 2024-05-08T00:44:40.785 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:44:40.777+0000 7fa00a173640 -1 reset not still connected to 0x556501bfa270 2024-05-08T00:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:44:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_24 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 journal 128M 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:44:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_24 128M 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_24 2024-05-08T00:44:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:44:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:44:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:44:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:44:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:44:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_24 2024-05-08T00:44:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:44:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:44:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=10 2024-05-08T00:44:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:44:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:44:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 42339328 bytes 2024-05-08T00:44:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 66014208 bytes 2024-05-08T00:44:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 92574720 bytes 2024-05-08T00:44:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 93210624 bytes 2024-05-08T00:44:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 97317888 bytes 2024-05-08T00:44:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 167568384 bytes 2024-05-08T00:44:53.513 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:53.513+0000 7f5317d5e640 -1 reset not still connected to 0x562084709450 2024-05-08T00:44:53.513 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:53.513+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_25 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 journal 128M 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_25 128M 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:44:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:44:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:44:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_25 2024-05-08T00:44:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:44:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:44:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:44:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:44:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:44:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_25 2024-05-08T00:44:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:44:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:44:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:44:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:44:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:44:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:44:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:44:57.845 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:57.845+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:44:57.845 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:57.845+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:44:57.845 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:57.846+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:44:57.846 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:57.846+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:44:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 54028288 bytes 2024-05-08T00:44:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 54671360 bytes 2024-05-08T00:44:58.409 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:44:58.409+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:44:59.134 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:44:59.134+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:45:01.473 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:45:01.474+0000 7facdd19b640 -1 bdev(0x562e81f6ae00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 3 2024-05-08T00:45:01.475 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:45:01.476+0000 7facdc99a640 -1 bdev(0x562e81f6ae00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 7 2024-05-08T00:45:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 117232640 bytes 2024-05-08T00:45:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 138711040 bytes 2024-05-08T00:45:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_26 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 journal 128M 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:45:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_26 128M 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_26 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_26 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:45:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:45:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=5 2024-05-08T00:45:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:45:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:45:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 8655872 bytes 2024-05-08T00:45:05.199 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:05.200+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:45:05.202 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:05.201+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:45:05.204 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:05.204+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:45:05.205 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:05.204+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:45:07.249 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:07.248+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:45:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:45:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:45:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:45:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:45:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:45:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_27 2024-05-08T00:45:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 journal 128M 2024-05-08T00:45:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_27 128M 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:45:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:45:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:45:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:45:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_27 2024-05-08T00:45:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_27 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:45:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:45:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=8 2024-05-08T00:45:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:45:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:45:10.695 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.694+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f4a04e0 2024-05-08T00:45:10.695 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.694+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f7c0000 2024-05-08T00:45:10.695 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.696+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:45:10.698 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.699+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:45:10.698 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.699+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:45:10.698 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.699+0000 7fbb0a622640 -1 reset not still connected to 0x55a781903110 2024-05-08T00:45:10.699 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.699+0000 7fbb0a622640 -1 reset not still connected to 0x55a78c115ad0 2024-05-08T00:45:10.699 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.700+0000 7fbb0a622640 -1 reset not still connected to 0x55a79b683520 2024-05-08T00:45:10.699 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.700+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a0035d40 2024-05-08T00:45:10.699 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.700+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:45:10.699 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.700+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:45:10.699 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.700+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:45:10.699 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:10.700+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:45:11.016 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.017+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:45:11.023 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.023+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:45:11.023 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:45:11.023 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:45:11.023 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:45:11.024 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.025+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:45:11.024 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.025+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:45:11.024 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.025+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:45:11.707 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.708+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:45:11.711 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:11.712+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:45:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 52385792 bytes 2024-05-08T00:45:14.879 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:45:14.880+0000 7fa00a173640 -1 reset not still connected to 0x5564f16fd790 2024-05-08T00:45:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 95517696 bytes 2024-05-08T00:45:14.881 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:45:14.883+0000 7fa00a173640 -1 reset not still connected to 0x5564f19b6dd0 2024-05-08T00:45:14.882 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:45:14.883+0000 7fa00a173640 -1 reset not still connected to 0x55650cda6410 2024-05-08T00:45:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 128081920 bytes 2024-05-08T00:45:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:45:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_28 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 journal 128M 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:45:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_28 128M 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_28 2024-05-08T00:45:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_28 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:45:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:45:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=8 2024-05-08T00:45:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:45:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:45:18.046 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:18.047+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:45:18.046 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:18.047+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:45:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 71965696 bytes 2024-05-08T00:45:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 126943232 bytes 2024-05-08T00:45:24.733 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:24.734+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:45:24.734 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:24.734+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:45:24.734 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:24.734+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:45:24.736 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:24.735+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:45:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:45:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:45:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:45:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:45:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:45:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_29 2024-05-08T00:45:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 journal 128M 2024-05-08T00:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:45:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:45:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:45:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_29 128M 2024-05-08T00:45:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:45:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:45:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:45:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:45:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_29 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_29 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:45:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:45:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=6 2024-05-08T00:45:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:45:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:45:25.433 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:25.433+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:45:25.434 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:25.434+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:45:25.434 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:25.434+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:45:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 94743552 bytes 2024-05-08T00:45:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_30 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 journal 128M 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:45:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_30 128M 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:45:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:45:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_30 2024-05-08T00:45:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:45:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:45:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:45:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:45:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:45:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_30 2024-05-08T00:45:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:45:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:45:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=8 2024-05-08T00:45:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:45:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:45:31.167 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:45:31.168+0000 7fa00a173640 -1 reset not still connected to 0x5564fd97ca90 2024-05-08T00:45:31.167 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:45:31.168+0000 7fa00a173640 -1 reset not still connected to 0x556501bfa270 2024-05-08T00:45:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 37014528 bytes 2024-05-08T00:45:34.496 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:34.497+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:45:34.496 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:34.497+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:45:34.496 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:34.497+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:45:34.496 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:34.497+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:45:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 49874944 bytes 2024-05-08T00:45:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 80172032 bytes 2024-05-08T00:45:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:45:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:45:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:45:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:45:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:45:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_31 2024-05-08T00:45:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 journal 128M 2024-05-08T00:45:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_31 128M 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:45:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:45:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:45:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_31 2024-05-08T00:45:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:45:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:45:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:45:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:45:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:45:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_31 2024-05-08T00:45:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:45:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:45:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=7 2024-05-08T00:45:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:45:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:45:39.806 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:39.808+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:45:39.807 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:39.808+0000 7f2ee8664640 -1 reset not still connected to 0x56323cfb95f0 2024-05-08T00:45:39.807 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:39.808+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:45:39.807 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:39.808+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd6680 2024-05-08T00:45:39.807 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:39.808+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd68f0 2024-05-08T00:45:39.807 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:45:39.808+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:45:43.047 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:45:43.048+0000 7facebded640 -1 reset not still connected to 0x562e8beced00 2024-05-08T00:45:43.150 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:43.152+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:45:43.153 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:43.154+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:45:43.154 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:43.155+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:45:43.257 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.257+0000 7f5317d5e640 -1 reset not still connected to 0x562084709450 2024-05-08T00:45:43.257 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.257+0000 7f5317d5e640 -1 reset not still connected to 0x56208a740410 2024-05-08T00:45:43.257 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.257+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:45:43.259 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.259+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:45:43.260 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.261+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:45:43.260 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.261+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:45:43.260 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.261+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:45:43.260 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.261+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:45:43.261 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.263+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:45:43.261 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:43.263+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:45:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 64369664 bytes 2024-05-08T00:45:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 84236288 bytes 2024-05-08T00:45:45.552 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:45.554+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:45:45.553 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:45.554+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:45:45.553 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:45.554+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:45:45.553 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:45.555+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:45:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:45:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_32 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 journal 128M 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:45:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_32 128M 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_32 2024-05-08T00:45:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:45:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:45:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:45:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:45:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:45:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_32 2024-05-08T00:45:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:45:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:45:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=5 2024-05-08T00:45:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:45:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:45:48.141 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:48.141+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:45:48.145 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:48.147+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:45:48.145 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:48.147+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:45:48.145 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:48.147+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:45:48.145 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:48.147+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:45:48.146 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:48.147+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:45:48.146 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:48.147+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:45:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 70661120 bytes 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_33 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 journal 128M 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:45:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_33 128M 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:45:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_33 2024-05-08T00:45:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:45:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:45:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:45:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:45:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:45:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_33 2024-05-08T00:45:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:45:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:45:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:45:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:45:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:45:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:45:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:45:51.900 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.899+0000 7fbb0a622640 -1 reset not still connected to 0x55a77fde65b0 2024-05-08T00:45:51.900 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.899+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a270 2024-05-08T00:45:51.900 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.900+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:45:51.900 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.900+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:45:51.900 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.900+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:45:51.900 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.900+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:45:51.900 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.900+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:45:51.901 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.903+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:45:51.901 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.903+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:45:51.904 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.905+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:45:51.904 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.905+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:45:51.904 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.905+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:45:51.904 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.905+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:45:51.904 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.905+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:45:51.904 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.905+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:45:51.904 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.905+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:45:51.904 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:45:51.905+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:45:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 40691712 bytes 2024-05-08T00:45:55.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:55.246+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:45:55.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:55.247+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:45:55.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:55.247+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:45:56.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 62623744 bytes 2024-05-08T00:45:56.612 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.613+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:45:56.612 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.613+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:45:56.612 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.613+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:45:56.612 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.613+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:45:56.612 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.613+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:45:56.612 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.613+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:45:56.835 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.836+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:45:56.835 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.836+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:45:56.835 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.836+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:45:56.836 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.836+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:45:56.836 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.837+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:45:56.836 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.839+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:45:56.836 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:56.839+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:45:58.460 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:58.462+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b9ee0 2024-05-08T00:45:58.460 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:58.462+0000 7f5317d5e640 -1 reset not still connected to 0x562092abe8f0 2024-05-08T00:45:58.460 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:58.462+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:45:58.465 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:58.467+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:45:58.465 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:58.467+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:45:58.465 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:58.467+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:45:58.466 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:58.467+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:45:58.469 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:58.472+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:45:58.469 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:58.472+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:45:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 120255488 bytes 2024-05-08T00:45:59.681 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:59.683+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:45:59.684 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:59.685+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:45:59.686 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:59.688+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:45:59.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:59.690+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:45:59.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:59.690+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:45:59.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:59.690+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:45:59.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:59.690+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:45:59.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:59.690+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:45:59.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:45:59.690+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:46:01.317 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.319+0000 7fa00a173640 -1 reset not still connected to 0x5564ecdc9860 2024-05-08T00:46:01.317 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.319+0000 7fa00a173640 -1 reset not still connected to 0x5564ed977e10 2024-05-08T00:46:01.318 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.319+0000 7fa00a173640 -1 reset not still connected to 0x5564ee15fad0 2024-05-08T00:46:01.318 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.319+0000 7fa00a173640 -1 reset not still connected to 0x5564ee15fe10 2024-05-08T00:46:01.318 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.319+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270270 2024-05-08T00:46:01.318 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.319+0000 7fa00a173640 -1 reset not still connected to 0x5564ee8b68f0 2024-05-08T00:46:01.318 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.319+0000 7fa00a173640 -1 reset not still connected to 0x5564f4cb4c30 2024-05-08T00:46:01.318 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.319+0000 7fa00a173640 -1 reset not still connected to 0x5564fd97ca90 2024-05-08T00:46:01.318 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.319+0000 7fa00a173640 -1 reset not still connected to 0x556501bfa270 2024-05-08T00:46:01.844 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.846+0000 7fa00a173640 -1 reset not still connected to 0x5564ecdc9860 2024-05-08T00:46:01.847 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.850+0000 7fa00a173640 -1 reset not still connected to 0x5564ee15fe10 2024-05-08T00:46:01.850 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.853+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270270 2024-05-08T00:46:01.851 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.854+0000 7fa00a173640 -1 reset not still connected to 0x5564ee8b68f0 2024-05-08T00:46:01.853 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.855+0000 7fa00a173640 -1 reset not still connected to 0x5564f4cb4c30 2024-05-08T00:46:01.853 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.855+0000 7fa00a173640 -1 reset not still connected to 0x5564f92e0340 2024-05-08T00:46:01.853 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.855+0000 7fa00a173640 -1 reset not still connected to 0x5564fd97ca90 2024-05-08T00:46:01.853 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:01.855+0000 7fa00a173640 -1 reset not still connected to 0x556501bfa270 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_34 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 journal 128M 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_34 128M 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:46:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_34 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_34 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:46:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:46:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:46:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:46:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:46:04.169 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:04.171+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:46:04.170 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:04.173+0000 7f5317d5e640 -1 reset not still connected to 0x5620a08b21a0 2024-05-08T00:46:04.171 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:04.173+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:46:04.171 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:04.173+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:46:04.175 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:04.178+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:46:04.175 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:04.178+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:46:04.175 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:04.178+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:46:04.175 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:04.178+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:46:04.175 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:04.178+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:46:06.597 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.599+0000 7fa00a173640 -1 reset not still connected to 0x5564ee15fad0 2024-05-08T00:46:06.597 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.599+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270270 2024-05-08T00:46:06.597 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.599+0000 7fa00a173640 -1 reset not still connected to 0x5564f92e0340 2024-05-08T00:46:06.597 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.599+0000 7fa00a173640 -1 reset not still connected to 0x5564fd97ca90 2024-05-08T00:46:06.597 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.599+0000 7fa00a173640 -1 reset not still connected to 0x556501bfa270 2024-05-08T00:46:06.635 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.637+0000 7fa00a173640 -1 reset not still connected to 0x5564f1216d00 2024-05-08T00:46:06.635 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.637+0000 7fa00a173640 -1 reset not still connected to 0x5564f16fd790 2024-05-08T00:46:06.635 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.637+0000 7fa00a173640 -1 reset not still connected to 0x5564f19b6dd0 2024-05-08T00:46:06.635 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.637+0000 7fa00a173640 -1 reset not still connected to 0x5564fe7296c0 2024-05-08T00:46:06.635 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.638+0000 7fa00a173640 -1 reset not still connected to 0x5564ecb408f0 2024-05-08T00:46:06.635 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.638+0000 7fa00a173640 -1 reset not still connected to 0x5564ecb41930 2024-05-08T00:46:06.638 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:06.641+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271790 2024-05-08T00:46:06.830 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:06.833+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:46:06.832 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:06.834+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:46:06.837 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:06.838+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:46:06.837 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:06.838+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:46:06.837 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:06.838+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:46:06.837 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:06.838+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:46:06.837 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:06.838+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:46:06.837 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:06.838+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:46:07.720 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:07.723+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:46:07.720 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:07.723+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:46:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 79881216 bytes 2024-05-08T00:46:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 82590720 bytes 2024-05-08T00:46:08.381 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:08.383+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f7c0000 2024-05-08T00:46:08.382 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:08.385+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:46:08.382 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:08.385+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:46:08.384 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:08.387+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:46:08.384 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:08.387+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:46:08.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:08.387+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:46:08.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:08.387+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:46:08.385 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:08.387+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:46:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 91319296 bytes 2024-05-08T00:46:09.615 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:09.617+0000 7fa2f29c3640 -1 reset not still connected to 0x5640eabfc750 2024-05-08T00:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:46:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_35 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 journal 128M 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_35 128M 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_35 2024-05-08T00:46:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:46:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:46:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:46:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:46:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:46:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_35 2024-05-08T00:46:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:46:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:46:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=6 2024-05-08T00:46:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:46:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:46:12.337 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:12.340+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f0614820 2024-05-08T00:46:12.337 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:12.340+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f8995c70 2024-05-08T00:46:13.031 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:13.031+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:46:13.039 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:13.040+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:46:13.039 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:13.040+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:46:13.039 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:13.040+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:46:13.039 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:13.040+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:46:13.539 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:13.540+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:46:13.539 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:13.540+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:46:13.541 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:13.541+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:46:13.541 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:13.541+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:46:13.541 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:13.541+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:46:13.541 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:13.541+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:46:13.542 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:13.543+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:46:13.542 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:13.543+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:46:14.824 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:14.824+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:46:14.827 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:14.828+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:46:14.830 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:14.828+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:46:14.830 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:14.828+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:46:14.831 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:14.828+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:46:14.831 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:14.828+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:46:14.831 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:14.828+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:46:14.831 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:14.829+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:46:15.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 41314304 bytes 2024-05-08T00:46:15.896 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:15.899+0000 7fa2e3568640 -1 bdev(0x5640ce37ee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-05-08T00:46:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 56382464 bytes 2024-05-08T00:46:17.570 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:17.571+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:46:17.570 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:17.572+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:46:17.570 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:17.572+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:46:17.570 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:17.572+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:46:17.571 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:17.572+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:46:17.571 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:17.573+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:46:17.572 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:17.573+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:46:17.860 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:17.862+0000 7fa00a173640 -1 reset not still connected to 0x5564f16fd790 2024-05-08T00:46:17.862 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:17.865+0000 7fa00a173640 -1 reset not still connected to 0x5564f19b6dd0 2024-05-08T00:46:17.862 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:17.865+0000 7fa00a173640 -1 reset not still connected to 0x5564fe7296c0 2024-05-08T00:46:17.865 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:17.867+0000 7fa00a173640 -1 reset not still connected to 0x55650c7ddba0 2024-05-08T00:46:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:46:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:46:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:46:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:46:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:46:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_36 2024-05-08T00:46:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 journal 128M 2024-05-08T00:46:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_36 128M 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:46:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:46:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:46:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:46:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_36 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_36 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:46:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:46:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=12 2024-05-08T00:46:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:46:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:46:18.112 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:18.112+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:46:18.113 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:18.112+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:46:18.114 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:18.112+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:46:18.114 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:18.112+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:46:18.114 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:18.112+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:46:18.114 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:18.112+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:46:21.077 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:21.078+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:46:21.077 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:21.078+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:46:21.077 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:21.078+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:46:21.077 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:21.078+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:46:21.077 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:21.078+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:46:21.078 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:21.080+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:46:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 65419264 bytes 2024-05-08T00:46:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 97417216 bytes 2024-05-08T00:46:25.973 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:25.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640e9ffd450 2024-05-08T00:46:25.973 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:25.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640ea004750 2024-05-08T00:46:25.973 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:25.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640ed0a7110 2024-05-08T00:46:25.973 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:25.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640edd8e750 2024-05-08T00:46:25.973 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:25.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640eeb55380 2024-05-08T00:46:25.973 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:25.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f117eb60 2024-05-08T00:46:25.973 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:25.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f23055f0 2024-05-08T00:46:25.973 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:46:25.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f34e0b60 2024-05-08T00:46:26.462 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:26.465+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ee0 2024-05-08T00:46:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 159366144 bytes 2024-05-08T00:46:29.759 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:29.762+0000 7fa00a173640 -1 reset not still connected to 0x55650c7ddba0 2024-05-08T00:46:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:46:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:46:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:46:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:46:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:46:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_37 2024-05-08T00:46:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 journal 128M 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_37 128M 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:46:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:46:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:46:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:46:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_37 2024-05-08T00:46:30.214 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.217+0000 7f5317d5e640 -1 reset not still connected to 0x56208b731a00 2024-05-08T00:46:30.215 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:46:30.217 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x56209736a000 2024-05-08T00:46:30.217 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:46:30.217 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x5620a08b21a0 2024-05-08T00:46:30.217 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:46:30.218 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:46:30.218 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:46:30.218 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:46:30.218 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:46:30.218 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:46:30.218 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.219+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_37 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:46:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:46:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=5 2024-05-08T00:46:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:46:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:46:30.559 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.562+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:46:30.559 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.562+0000 7f5317d5e640 -1 reset not still connected to 0x56208b731a00 2024-05-08T00:46:30.559 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.562+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:46:30.561 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.563+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:46:30.561 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.563+0000 7f5317d5e640 -1 reset not still connected to 0x5620a08b21a0 2024-05-08T00:46:30.561 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.563+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:46:30.561 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.563+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:46:30.561 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.563+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:46:30.561 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.563+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:46:30.561 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.563+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:46:30.561 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:30.563+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:46:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 614400 bytes 2024-05-08T00:46:32.415 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:32.418+0000 7fa00a173640 -1 reset not still connected to 0x5564ecb41c70 2024-05-08T00:46:32.416 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:32.418+0000 7fa00a173640 -1 reset not still connected to 0x5564eeeae340 2024-05-08T00:46:32.416 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:32.418+0000 7fa00a173640 -1 reset not still connected to 0x5564f1216d00 2024-05-08T00:46:32.416 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:32.418+0000 7fa00a173640 -1 reset not still connected to 0x5564f19b6dd0 2024-05-08T00:46:32.416 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:32.418+0000 7fa00a173640 -1 reset not still connected to 0x5564fe7296c0 2024-05-08T00:46:32.416 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:32.418+0000 7fa00a173640 -1 reset not still connected to 0x55650c7ddba0 2024-05-08T00:46:33.794 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:33.797+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:46:33.795 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:33.797+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:46:33.795 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:33.797+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:46:33.795 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:33.797+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_38 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 journal 128M 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_38 128M 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_38 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_38 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:46:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:46:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=7 2024-05-08T00:46:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:46:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:46:37.246 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.249+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:46:37.246 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.249+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:46:37.255 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.257+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:46:37.255 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.257+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:46:37.255 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.257+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:46:37.255 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.257+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:46:37.255 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.257+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:46:37.256 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.258+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:46:37.256 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.258+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:46:37.256 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.258+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:46:37.256 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.258+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a103f6c0 2024-05-08T00:46:37.256 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.258+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:46:37.256 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.258+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:46:37.256 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.258+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:46:37.256 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.258+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:46:37.256 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:37.258+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:46:39.513 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.511+0000 7f5317d5e640 -1 reset not still connected to 0x562092abe8f0 2024-05-08T00:46:39.513 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.511+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:46:39.513 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.513+0000 7f5317d5e640 -1 reset not still connected to 0x56209393a820 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.513+0000 7f5317d5e640 -1 reset not still connected to 0x562099e6ad00 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.513+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.513+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.513+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.513+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.513+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.513+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.513+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.515+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:46:39.514 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:39.515+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:46:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 74745856 bytes 2024-05-08T00:46:41.835 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:41.831+0000 7f9ff6d10640 -1 bdev(0x5564eb38ce00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-05-08T00:46:41.840 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:41.841+0000 7f9ff5d0e640 -1 bdev(0x5564eb38ce00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-05-08T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:46:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_39 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 journal 128M 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_39 128M 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_39 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_39 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:46:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:46:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=5 2024-05-08T00:46:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:46:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:46:45.146 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:45.149+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:46:45.146 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:45.149+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:46:45.146 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:45.149+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:46:45.146 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:45.149+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:46:45.146 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:45.149+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:46:47.301 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:46:47.304+0000 7facdb998640 -1 bdev(0x562e81f6ae00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 2 2024-05-08T00:46:47.304 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:46:47.307+0000 7facdc99a640 -1 bdev(0x562e81f6ae00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 11 2024-05-08T00:46:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 76158976 bytes 2024-05-08T00:46:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:46:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:46:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:46:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_40 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 journal 128M 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_40 128M 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:46:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:46:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:46:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:46:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_40 2024-05-08T00:46:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:46:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:46:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:46:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:46:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:46:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_40 2024-05-08T00:46:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:46:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:46:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:46:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:46:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:46:48.003 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:46:48.006+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:46:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 19224576 bytes 2024-05-08T00:46:49.217 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:49.221+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cca90 2024-05-08T00:46:49.222 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:49.226+0000 7f2ee8664640 -1 reset not still connected to 0x56321c121110 2024-05-08T00:46:49.222 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:49.226+0000 7f2ee8664640 -1 reset not still connected to 0x56321c336a90 2024-05-08T00:46:49.222 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:49.226+0000 7f2ee8664640 -1 reset not still connected to 0x5632319b6f70 2024-05-08T00:46:49.228 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:49.232+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:46:49.228 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:49.232+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:46:49.228 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:49.232+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:46:49.229 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:49.232+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:46:49.229 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:49.232+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:46:51.417 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:51.418+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270410 2024-05-08T00:46:51.418 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:51.419+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271110 2024-05-08T00:46:51.418 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:51.419+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271790 2024-05-08T00:46:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 69550080 bytes 2024-05-08T00:46:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 95736832 bytes 2024-05-08T00:46:53.794 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:53.798+0000 7f2ee8664640 -1 reset not still connected to 0x56321c336a90 2024-05-08T00:46:53.797 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:53.799+0000 7f2ee8664640 -1 reset not still connected to 0x5632319b6f70 2024-05-08T00:46:53.800 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:53.804+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:46:53.800 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:53.804+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:46:53.800 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:53.804+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:46:53.800 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:53.804+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:46:53.800 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:53.804+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:46:53.800 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:53.804+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:46:55.972 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:55.975+0000 7f2ee8664640 -1 reset not still connected to 0x56321c121110 2024-05-08T00:46:55.972 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:55.975+0000 7f2ee8664640 -1 reset not still connected to 0x56321c336a90 2024-05-08T00:46:55.973 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:55.975+0000 7f2ee8664640 -1 reset not still connected to 0x5632319b6f70 2024-05-08T00:46:55.973 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:55.975+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:46:55.973 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:55.975+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:46:55.973 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:55.975+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:46:55.973 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:46:55.975+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:46:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:46:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_41 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 journal 128M 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_41 128M 2024-05-08T00:46:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:46:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_41 2024-05-08T00:46:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_41 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:46:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:46:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=5 2024-05-08T00:46:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:46:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:46:56.958 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.961+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:46:56.958 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.961+0000 7f5317d5e640 -1 reset not still connected to 0x56208b731a00 2024-05-08T00:46:56.958 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.961+0000 7f5317d5e640 -1 reset not still connected to 0x562091a9ad00 2024-05-08T00:46:56.958 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.962+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:46:56.959 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.963+0000 7f5317d5e640 -1 reset not still connected to 0x56209736a000 2024-05-08T00:46:56.959 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.963+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:46:56.961 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.964+0000 7f5317d5e640 -1 reset not still connected to 0x5620a08b21a0 2024-05-08T00:46:56.962 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.964+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:46:56.962 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.964+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:46:56.962 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.964+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:46:56.962 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.964+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:46:56.962 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.964+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:46:56.964 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.968+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:46:56.964 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:56.968+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:46:57.020 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.024+0000 7f5317d5e640 -1 reset not still connected to 0x562082e12dd0 2024-05-08T00:46:57.020 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.024+0000 7f5317d5e640 -1 reset not still connected to 0x562084709450 2024-05-08T00:46:57.026 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.027+0000 7f5317d5e640 -1 reset not still connected to 0x56208a740410 2024-05-08T00:46:57.026 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.027+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:46:57.026 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.027+0000 7f5317d5e640 -1 reset not still connected to 0x562091a9ad00 2024-05-08T00:46:57.026 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.027+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:46:57.026 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.027+0000 7f5317d5e640 -1 reset not still connected to 0x56209736a000 2024-05-08T00:46:57.026 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.028+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:46:57.027 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.031+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:46:57.027 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.031+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:46:57.027 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.031+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:46:57.028 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.031+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:46:57.028 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.031+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:46:57.029 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.033+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:46:57.030 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:46:57.033+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:46:58.914 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:58.918+0000 7fa00a173640 -1 reset not still connected to 0x5564f19b6dd0 2024-05-08T00:46:58.914 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:58.918+0000 7fa00a173640 -1 reset not still connected to 0x5564fe7296c0 2024-05-08T00:46:58.914 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:58.918+0000 7fa00a173640 -1 reset not still connected to 0x55650c7ddba0 2024-05-08T00:46:58.914 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:46:58.918+0000 7fa00a173640 -1 reset not still connected to 0x556511d27ee0 2024-05-08T00:46:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 35646464 bytes 2024-05-08T00:47:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 47182848 bytes 2024-05-08T00:47:01.384 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:01.385+0000 7fa2f29c3640 -1 reset not still connected to 0x5640d190af70 2024-05-08T00:47:01.385 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:01.385+0000 7fa2f29c3640 -1 reset not still connected to 0x5640e19a6750 2024-05-08T00:47:01.385 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:01.390+0000 7fa2f29c3640 -1 reset not still connected to 0x5640e9ffd450 2024-05-08T00:47:01.388 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:01.391+0000 7fa2f29c3640 -1 reset not still connected to 0x5640ea004750 2024-05-08T00:47:01.388 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:01.391+0000 7fa2f29c3640 -1 reset not still connected to 0x5640ed0a7110 2024-05-08T00:47:01.388 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:01.391+0000 7fa2f29c3640 -1 reset not still connected to 0x5640edd8e750 2024-05-08T00:47:01.388 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:01.391+0000 7fa2f29c3640 -1 reset not still connected to 0x5640eeb55380 2024-05-08T00:47:01.388 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:01.391+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f23055f0 2024-05-08T00:47:01.388 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:01.391+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f34e0b60 2024-05-08T00:47:01.429 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.432+0000 7f2ee8664640 -1 reset not still connected to 0x563219425040 2024-05-08T00:47:01.429 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.432+0000 7f2ee8664640 -1 reset not still connected to 0x56321c121110 2024-05-08T00:47:01.429 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.432+0000 7f2ee8664640 -1 reset not still connected to 0x56321c336a90 2024-05-08T00:47:01.429 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.432+0000 7f2ee8664640 -1 reset not still connected to 0x5632319b6f70 2024-05-08T00:47:01.429 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.432+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:47:01.430 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.434+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:47:01.430 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.434+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:47:01.430 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.434+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:47:01.430 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.434+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:47:01.431 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:01.434+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:47:01.587 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:47:01.590+0000 7facebded640 -1 reset not still connected to 0x562ea1345e10 2024-05-08T00:47:01.587 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:47:01.590+0000 7facebded640 -1 reset not still connected to 0x562eaa8ddc70 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_42 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 journal 128M 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_42 128M 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:47:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:47:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:47:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_42 2024-05-08T00:47:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:47:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:47:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:47:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:47:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:47:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_42 2024-05-08T00:47:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:47:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:47:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:47:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:47:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:47:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 38288384 bytes 2024-05-08T00:47:06.675 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:47:06.678+0000 7facebded640 -1 reset not still connected to 0x562e83926ea0 2024-05-08T00:47:06.816 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.819+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:47:06.817 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.819+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23380 2024-05-08T00:47:06.817 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.819+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23c70 2024-05-08T00:47:06.817 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.819+0000 7fbb0a622640 -1 reset not still connected to 0x55a781903110 2024-05-08T00:47:06.817 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.820+0000 7fbb0a622640 -1 reset not still connected to 0x55a78c115ad0 2024-05-08T00:47:06.817 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a795fcb1e0 2024-05-08T00:47:06.819 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:47:06.820 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a79b683520 2024-05-08T00:47:06.820 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a0035d40 2024-05-08T00:47:06.820 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06c5110 2024-05-08T00:47:06.820 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:47:06.820 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:47:06.820 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:47:06.820 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:47:06.820 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f4a04e0 2024-05-08T00:47:06.820 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:06.821+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23110 2024-05-08T00:47:07.164 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:07.165+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:47:07.164 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:07.165+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:47:07.165 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:07.165+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:47:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 136008704 bytes 2024-05-08T00:47:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 137406464 bytes 2024-05-08T00:47:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:47:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:47:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_43 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 journal 128M 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_43 128M 2024-05-08T00:47:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_43 2024-05-08T00:47:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:47:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:47:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:47:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:47:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:47:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_43 2024-05-08T00:47:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:47:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:47:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=13 2024-05-08T00:47:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:47:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:47:12.888 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:12.888+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:47:12.888 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:12.888+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:47:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 14330880 bytes 2024-05-08T00:47:13.880 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:13.885+0000 7fa2f29c3640 -1 reset not still connected to 0x5640eabfc750 2024-05-08T00:47:13.880 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:47:13.885+0000 7fa2f29c3640 -1 reset not still connected to 0x5640ecaf7450 2024-05-08T00:47:18.315 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:18.319+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:47:18.316 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:18.319+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:47:18.316 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:18.319+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:47:18.319 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:18.321+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:47:18.319 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:18.324+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:47:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 83602432 bytes 2024-05-08T00:47:20.511 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.516+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:47:20.513 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.518+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:47:20.515 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.519+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:47:20.515 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.519+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:47:20.515 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.519+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:47:20.515 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.519+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:47:20.515 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.519+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:47:20.515 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.519+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:47:20.515 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.519+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:47:20.516 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.519+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:47:20.516 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.519+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b7f7d1e0 2024-05-08T00:47:20.518 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.520+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:47:20.520 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:20.523+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:47:20.821 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:20.825+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:47:20.821 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:20.825+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:47:20.821 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:20.825+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:47:20.821 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:20.825+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:47:20.821 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:20.825+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:47:20.821 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:20.825+0000 7f2ee8664640 -1 reset not still connected to 0x56321c336a90 2024-05-08T00:47:20.822 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:20.825+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:47:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 121405440 bytes 2024-05-08T00:47:22.111 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.112+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:47:22.113 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.116+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:47:22.113 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.116+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:47:22.113 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.116+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:47:22.113 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.116+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:47:22.113 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.116+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:47:22.116 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.120+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:47:22.116 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.120+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a103f6c0 2024-05-08T00:47:22.116 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.120+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:47:22.116 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.120+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:47:22.116 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.120+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:47:22.116 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.120+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:47:22.116 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.120+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b7f7d1e0 2024-05-08T00:47:22.119 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.124+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:47:22.119 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:22.124+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:47:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 169684992 bytes 2024-05-08T00:47:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:47:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:47:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_44 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 journal 128M 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_44 128M 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:47:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:47:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:47:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:47:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_44 2024-05-08T00:47:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:47:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:47:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:47:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:47:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:47:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_44 2024-05-08T00:47:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:47:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:47:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=5 2024-05-08T00:47:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:47:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:47:26.175 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.178+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:47:26.175 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.178+0000 7f5317d5e640 -1 reset not still connected to 0x562091a9ad00 2024-05-08T00:47:26.179 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.183+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:47:26.179 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.183+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:47:26.180 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620a08b21a0 2024-05-08T00:47:26.180 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:47:26.180 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:47:26.181 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:47:26.181 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:47:26.181 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:47:26.181 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:47:26.181 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:47:26.181 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:26.185+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:47:26.248 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.250+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:47:26.249 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:47:26.249 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:47:26.249 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a103f6c0 2024-05-08T00:47:26.250 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:47:26.250 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:47:26.250 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:47:26.250 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:47:26.250 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b7f7d1e0 2024-05-08T00:47:26.250 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:47:26.250 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:26.253+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_45 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 journal 128M 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_45 128M 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:47:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:47:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:47:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_45 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_45 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:47:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:47:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=12 2024-05-08T00:47:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:47:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:47:32.236 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.240+0000 7f5317d5e640 -1 reset not still connected to 0x562092abe8f0 2024-05-08T00:47:32.236 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.240+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:47:32.238 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.242+0000 7f5317d5e640 -1 reset not still connected to 0x56209393a820 2024-05-08T00:47:32.240 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.245+0000 7f5317d5e640 -1 reset not still connected to 0x562099e6ad00 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x56209ddcc680 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620a57cc750 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:47:32.245 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:47:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b8d00 2024-05-08T00:47:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x562092abe8f0 2024-05-08T00:47:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:47:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:47:32.246 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:32.249+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:47:33.127 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.129+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:47:33.127 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.130+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a0035d40 2024-05-08T00:47:33.127 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.130+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06229c0 2024-05-08T00:47:33.127 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.130+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a65c52b0 2024-05-08T00:47:33.127 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.130+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:47:33.129 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.134+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:47:33.129 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.134+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:47:33.129 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.134+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:47:33.939 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a77fde65b0 2024-05-08T00:47:33.939 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a270 2024-05-08T00:47:33.939 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:47:33.939 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:47:33.940 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:47:33.941 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:47:33.941 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:47:33.941 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:47:33.941 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:47:33.942 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.945+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:47:33.942 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.945+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:47:33.942 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.945+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:47:33.942 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.945+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:47:33.942 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.945+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a103f6c0 2024-05-08T00:47:33.942 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.945+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:47:33.942 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.945+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:47:33.943 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.947+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:47:33.943 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.947+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:47:33.944 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.949+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b7f7d1e0 2024-05-08T00:47:33.944 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.949+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:47:33.944 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:33.949+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:47:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 57944064 bytes 2024-05-08T00:47:36.985 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:36.986+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:47:36.985 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:36.986+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:47:36.986 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:36.986+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:47:36.986 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:36.989+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b7f7d1e0 2024-05-08T00:47:36.986 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:36.989+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:47:36.986 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:36.989+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:47:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 106822656 bytes 2024-05-08T00:47:38.686 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.690+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:47:38.687 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x56209393a820 2024-05-08T00:47:38.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x562099e6ad00 2024-05-08T00:47:38.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:47:38.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:47:38.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:47:38.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x5620a57cc750 2024-05-08T00:47:38.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:47:38.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:47:38.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:47:38.688 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.692+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:47:38.689 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.695+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:47:38.690 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:38.695+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:47:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 145021952 bytes 2024-05-08T00:47:40.444 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:40.449+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:47:40.446 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:40.450+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:47:40.450 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:40.455+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:47:40.457 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:40.461+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:47:40.458 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:40.462+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:47:41.064 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.064+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:47:41.064 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.064+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:47:41.064 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.065+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:47:41.064 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.065+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:47:41.066 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.070+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:47:41.066 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.070+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:47:41.066 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.070+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:47:41.066 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.070+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:47:41.066 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.070+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:47:41.066 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.070+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:47:41.066 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.070+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a103f6c0 2024-05-08T00:47:41.066 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.072+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:47:41.068 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.072+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:47:41.068 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.073+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:47:41.068 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.073+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:47:41.068 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.073+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b7f7d1e0 2024-05-08T00:47:41.068 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.073+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:47:41.068 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:41.073+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:47:41.418 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.419+0000 7f5317d5e640 -1 reset not still connected to 0x56209ddcc680 2024-05-08T00:47:41.419 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.424+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:47:41.422 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.427+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:47:41.422 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.427+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:47:41.422 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.427+0000 7f5317d5e640 -1 reset not still connected to 0x5620a57cc750 2024-05-08T00:47:41.427 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.432+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:47:41.427 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.432+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:47:41.427 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.432+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:47:41.427 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.432+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:47:41.427 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.432+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:47:41.427 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.432+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:47:41.427 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:41.432+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:47:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 159506432 bytes 2024-05-08T00:47:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_46 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 journal 128M 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:47:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_46 128M 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_46 2024-05-08T00:47:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:47:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:47:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:47:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:47:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:47:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_46 2024-05-08T00:47:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:47:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:47:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:47:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:47:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:47:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 68919296 bytes 2024-05-08T00:47:48.978 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:48.983+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:47:48.980 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:48.985+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:47:48.980 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:48.986+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:47:48.980 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:48.986+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:47:49.477 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.481+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b9ee0 2024-05-08T00:47:49.483 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.488+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:47:49.483 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.488+0000 7f5317d5e640 -1 reset not still connected to 0x562092abe8f0 2024-05-08T00:47:49.483 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.488+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:47:49.483 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.488+0000 7f5317d5e640 -1 reset not still connected to 0x56209393a820 2024-05-08T00:47:49.483 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.488+0000 7f5317d5e640 -1 reset not still connected to 0x562099e6ad00 2024-05-08T00:47:49.483 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.488+0000 7f5317d5e640 -1 reset not still connected to 0x56209ddcc680 2024-05-08T00:47:49.483 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.488+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:47:49.486 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.491+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:47:49.486 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.491+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:47:49.486 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.491+0000 7f5317d5e640 -1 reset not still connected to 0x5620a57cc750 2024-05-08T00:47:49.486 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.491+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:47:49.486 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.491+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:47:49.488 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.493+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:47:49.488 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.493+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:47:49.488 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.493+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:47:49.488 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.493+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:47:49.488 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:49.493+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:47:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 135735296 bytes 2024-05-08T00:47:52.044 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x56209ddcc680 2024-05-08T00:47:52.044 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:47:52.044 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:47:52.044 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:47:52.044 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620a57cc750 2024-05-08T00:47:52.044 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:47:52.044 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:47:52.044 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:47:52.045 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:47:52.045 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:47:52.045 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:47:52.045 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:52.048+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:47:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_47 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 journal 128M 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_47 128M 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:47:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:47:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_47 2024-05-08T00:47:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:47:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:47:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:47:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:47:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:47:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_47 2024-05-08T00:47:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:47:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:47:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:47:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:47:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=8 2024-05-08T00:47:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:47:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:47:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 13156352 bytes 2024-05-08T00:47:54.041 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x563226151a00 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x563229ab1ad0 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x56323cfb95f0 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x56323d171d40 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x563243799ad0 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd6680 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd68f0 2024-05-08T00:47:54.042 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:54.045+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:47:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 28223488 bytes 2024-05-08T00:47:55.140 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.145+0000 7f2ee8664640 -1 reset not still connected to 0x56321c121110 2024-05-08T00:47:55.147 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.151+0000 7f2ee8664640 -1 reset not still connected to 0x56321c336a90 2024-05-08T00:47:55.147 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.152+0000 7f2ee8664640 -1 reset not still connected to 0x5632319b6f70 2024-05-08T00:47:55.147 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.152+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:47:55.147 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.152+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:47:55.148 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.152+0000 7f2ee8664640 -1 reset not still connected to 0x56323870a1a0 2024-05-08T00:47:55.148 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.152+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:47:55.148 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.152+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:47:55.148 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.152+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:47:55.148 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:47:55.152+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:47:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 57993216 bytes 2024-05-08T00:47:58.345 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.349+0000 7f5317d5e640 -1 reset not still connected to 0x562091a9ad00 2024-05-08T00:47:58.348 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.351+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:47:58.348 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.352+0000 7f5317d5e640 -1 reset not still connected to 0x56209736a000 2024-05-08T00:47:58.348 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.353+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:47:58.350 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.355+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:47:58.350 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.355+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:47:58.350 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.355+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:47:58.350 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.355+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:47:58.350 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.355+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:47:58.351 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.355+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:47:58.351 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.355+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:47:58.356 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:58.361+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:47:59.162 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.166+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:47:59.207 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.168+0000 7f5317d5e640 -1 reset not still connected to 0x562091a9ad00 2024-05-08T00:47:59.207 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.176+0000 7f5317d5e640 -1 reset not still connected to 0x56209736a000 2024-05-08T00:47:59.207 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.180+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:47:59.207 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.180+0000 7f5317d5e640 -1 reset not still connected to 0x56209d36fd40 2024-05-08T00:47:59.207 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.182+0000 7f5317d5e640 -1 reset not still connected to 0x5620a08b21a0 2024-05-08T00:47:59.207 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.182+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:47:59.207 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.182+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:47:59.208 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.182+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:47:59.208 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.182+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:47:59.208 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.182+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:47:59.208 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.182+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:47:59.208 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.184+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:47:59.208 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:47:59.186+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:47:59.289 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:59.294+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:47:59.302 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:59.307+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:47:59.302 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:47:59.307+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:47:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 117556224 bytes 2024-05-08T00:48:00.226 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.226+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0ba00 2024-05-08T00:48:00.226 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.226+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:48:00.226 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.226+0000 7f2ee8664640 -1 reset not still connected to 0x56321c360c30 2024-05-08T00:48:00.226 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.226+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:48:00.226 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.226+0000 7f2ee8664640 -1 reset not still connected to 0x563226151a00 2024-05-08T00:48:00.226 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.230+0000 7f2ee8664640 -1 reset not still connected to 0x563229ab1ad0 2024-05-08T00:48:00.226 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.230+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:48:00.226 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.230+0000 7f2ee8664640 -1 reset not still connected to 0x563235842d00 2024-05-08T00:48:00.228 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.232+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:48:00.229 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.232+0000 7f2ee8664640 -1 reset not still connected to 0x56323cfb95f0 2024-05-08T00:48:00.229 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.232+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:48:00.229 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.232+0000 7f2ee8664640 -1 reset not still connected to 0x56323d171d40 2024-05-08T00:48:00.229 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.233+0000 7f2ee8664640 -1 reset not still connected to 0x563243799ad0 2024-05-08T00:48:00.229 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.233+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd68f0 2024-05-08T00:48:00.229 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:00.233+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:48:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:48:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:48:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:48:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_48 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 journal 128M 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_48 128M 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:48:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:48:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:48:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:48:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_48 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_48 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:48:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:48:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=5 2024-05-08T00:48:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:48:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:48:00.612 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.615+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:48:00.613 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.617+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:48:00.615 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.619+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:48:00.617 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.619+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:48:00.617 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.619+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:48:00.617 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.619+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:48:00.738 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.742+0000 7f5317d5e640 -1 reset not still connected to 0x5620832b8d00 2024-05-08T00:48:00.742 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.747+0000 7f5317d5e640 -1 reset not still connected to 0x56208b200680 2024-05-08T00:48:00.745 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.750+0000 7f5317d5e640 -1 reset not still connected to 0x562092abe8f0 2024-05-08T00:48:00.745 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.750+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:48:00.750 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.754+0000 7f5317d5e640 -1 reset not still connected to 0x56209393a820 2024-05-08T00:48:00.750 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.754+0000 7f5317d5e640 -1 reset not still connected to 0x562099e6ad00 2024-05-08T00:48:00.750 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.754+0000 7f5317d5e640 -1 reset not still connected to 0x56209ddcc680 2024-05-08T00:48:00.751 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.754+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:48:00.751 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.754+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:48:00.751 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.754+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:48:00.751 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.754+0000 7f5317d5e640 -1 reset not still connected to 0x5620a57cc750 2024-05-08T00:48:00.751 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.754+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:48:00.751 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.754+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:48:00.754 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.757+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:48:00.754 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.757+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:48:00.755 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.757+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:48:00.755 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.757+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:48:00.755 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:00.757+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:48:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 6773760 bytes 2024-05-08T00:48:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 55858176 bytes 2024-05-08T00:48:03.154 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:03.159+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0ba00 2024-05-08T00:48:03.160 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:03.162+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:48:03.160 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:03.162+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:48:03.160 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:03.162+0000 7f2ee8664640 -1 reset not still connected to 0x56323cfb95f0 2024-05-08T00:48:03.160 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:03.162+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:48:03.939 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:03.944+0000 7fa2f29c3640 -1 reset not still connected to 0x5640e9ffd450 2024-05-08T00:48:03.939 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:03.944+0000 7fa2f29c3640 -1 reset not still connected to 0x5640ea004750 2024-05-08T00:48:03.939 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:03.944+0000 7fa2f29c3640 -1 reset not still connected to 0x5640ed0a7110 2024-05-08T00:48:03.939 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:03.944+0000 7fa2f29c3640 -1 reset not still connected to 0x5640eeb55380 2024-05-08T00:48:03.939 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:03.944+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f23055f0 2024-05-08T00:48:03.939 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:03.944+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f34e0b60 2024-05-08T00:48:03.940 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:03.944+0000 7fa2f29c3640 -1 reset not still connected to 0x5640fdbdf520 2024-05-08T00:48:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:48:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:48:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:48:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_49 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 journal 128M 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:48:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_49 128M 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_49 2024-05-08T00:48:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:48:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:48:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:48:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:48:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:48:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_49 2024-05-08T00:48:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:48:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:48:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=7 2024-05-08T00:48:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:48:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:48:09.581 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:09.587+0000 7fa00a173640 -1 reset not still connected to 0x5564ecb415f0 2024-05-08T00:48:09.585 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:09.590+0000 7fa00a173640 -1 reset not still connected to 0x5564f16fd790 2024-05-08T00:48:09.585 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:09.590+0000 7fa00a173640 -1 reset not still connected to 0x5564f19b6dd0 2024-05-08T00:48:09.586 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:09.590+0000 7fa00a173640 -1 reset not still connected to 0x5564ecb415f0 2024-05-08T00:48:11.496 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:11.501+0000 7fbb0a622640 -1 reset not still connected to 0x55a795fcb1e0 2024-05-08T00:48:11.496 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:11.501+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:48:11.496 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:11.501+0000 7fbb0a622640 -1 reset not still connected to 0x55a79b683520 2024-05-08T00:48:11.496 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:11.501+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06229c0 2024-05-08T00:48:11.496 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:11.501+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a65c52b0 2024-05-08T00:48:11.496 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:11.501+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:48:11.497 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:11.501+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:48:11.497 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:11.501+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:48:11.497 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:11.501+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:48:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_50 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 journal 128M 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:48:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n journal ']' 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_50 128M 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 128M ']' 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=128M 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_50 2024-05-08T00:48:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ is_pool_mirror_mode_image mirror 2024-05-08T00:48:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' pool = image ']' 2024-05-08T00:48:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ case "${pool}" in 2024-05-08T00:48:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 1 2024-05-08T00:48:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n '' ']' 2024-05-08T00:48:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_50 2024-05-08T00:48:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:48:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:48:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=13 2024-05-08T00:48:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:48:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:48:14.983 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:14.989+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:48:14.983 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:14.989+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:48:14.983 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:14.989+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:48:16.428 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:16.434+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271790 2024-05-08T00:48:16.744 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.750+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:48:16.749 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.755+0000 7f2ee8664640 -1 reset not still connected to 0x5632159288f0 2024-05-08T00:48:16.750 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.755+0000 7f2ee8664640 -1 reset not still connected to 0x56321c360c30 2024-05-08T00:48:16.750 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.755+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:48:16.750 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.755+0000 7f2ee8664640 -1 reset not still connected to 0x563226151a00 2024-05-08T00:48:16.750 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.755+0000 7f2ee8664640 -1 reset not still connected to 0x563229ab1ad0 2024-05-08T00:48:16.750 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:48:16.751 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x563235842d00 2024-05-08T00:48:16.751 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:48:16.751 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x56323cfb95f0 2024-05-08T00:48:16.751 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:48:16.751 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x56323d171d40 2024-05-08T00:48:16.751 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x563243799ad0 2024-05-08T00:48:16.751 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd6680 2024-05-08T00:48:16.751 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd68f0 2024-05-08T00:48:16.751 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:16.756+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:48:17.969 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:17.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640edd8e750 2024-05-08T00:48:17.969 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:17.975+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f23055f0 2024-05-08T00:48:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 91845632 bytes 2024-05-08T00:48:19.934 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:19.940+0000 7f9ff8d14640 -1 bdev(0x5564eb38ce00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2024-05-08T00:48:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 137166848 bytes 2024-05-08T00:48:21.676 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:21.679+0000 7fa00a173640 -1 reset not still connected to 0x5564f16fd790 2024-05-08T00:48:21.676 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:21.679+0000 7fa00a173640 -1 reset not still connected to 0x5564fe7296c0 2024-05-08T00:48:21.676 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:21.679+0000 7fa00a173640 -1 reset not still connected to 0x55650c7ddba0 2024-05-08T00:48:21.676 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:21.679+0000 7fa00a173640 -1 reset not still connected to 0x556511d27ee0 2024-05-08T00:48:21.676 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:21.679+0000 7fa00a173640 -1 reset not still connected to 0x556513e4fd40 2024-05-08T00:48:21.676 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:21.679+0000 7fa00a173640 -1 reset not still connected to 0x5564ecb41930 2024-05-08T00:48:21.676 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:21.679+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270410 2024-05-08T00:48:21.676 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:21.679+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271110 2024-05-08T00:48:21.677 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:21.679+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271790 2024-05-08T00:48:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 166144000 bytes 2024-05-08T00:48:23.650 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:48:23.653+0000 7facebded640 -1 reset not still connected to 0x562ea274e820 2024-05-08T00:48:24.461 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.463+0000 7f5317d5e640 -1 reset not still connected to 0x562082e12dd0 2024-05-08T00:48:24.462 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.463+0000 7f5317d5e640 -1 reset not still connected to 0x562084709450 2024-05-08T00:48:24.462 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.463+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:48:24.463 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.465+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:48:24.465 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.465+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:48:24.465 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.465+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:48:24.466 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.472+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:48:24.467 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.472+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:48:24.467 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.472+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:48:24.473 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.478+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:48:24.474 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.478+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:48:24.474 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:24.478+0000 7f5317d5e640 -1 reset not still connected to 0x5620bdda1110 2024-05-08T00:48:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:48:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:48:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_count=50 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local count 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_count=0 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 40 2024-05-08T00:48:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in `seq 1 40` 2024-05-08T00:48:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 1 -ne 1 2024-05-08T00:48:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-05-08T00:48:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep 'images: ' 2024-05-08T00:48:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut '-d ' -f 2 2024-05-08T00:48:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: failed to query services: (13) Permission denied 2024-05-08T00:48:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ count=50 2024-05-08T00:48:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 50 = 50 2024-05-08T00:48:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-05-08T00:48:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:48:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state 2024-05-08T00:48:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 40 2024-05-08T00:48:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in `seq 1 40` 2024-05-08T00:48:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 1 -ne 1 2024-05-08T00:48:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-05-08T00:48:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep 'image health:' 2024-05-08T00:48:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut '-d ' -f 3 2024-05-08T00:48:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: failed to query services: (13) Permission denied 2024-05-08T00:48:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=WARNING 2024-05-08T00:48:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test WARNING = ERROR 2024-05-08T00:48:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test WARNING = OK 2024-05-08T00:48:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in `seq 1 40` 2024-05-08T00:48:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 2 -ne 1 2024-05-08T00:48:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 30 2024-05-08T00:48:26.111 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:26.116+0000 7fa00a173640 -1 reset not still connected to 0x5564ee15fad0 2024-05-08T00:48:26.111 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:26.118+0000 7fa00a173640 -1 reset not still connected to 0x5564ee15fe10 2024-05-08T00:48:26.111 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:26.118+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270270 2024-05-08T00:48:26.112 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:26.118+0000 7fa00a173640 -1 reset not still connected to 0x5564f90840d0 2024-05-08T00:48:26.112 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:26.118+0000 7fa00a173640 -1 reset not still connected to 0x5564f92e0340 2024-05-08T00:48:26.112 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:26.118+0000 7fa00a173640 -1 reset not still connected to 0x5564fd97ca90 2024-05-08T00:48:26.112 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:26.118+0000 7fa00a173640 -1 reset not still connected to 0x556501bfa270 2024-05-08T00:48:26.761 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:26.767+0000 7f2ee8664640 -1 reset not still connected to 0x56323870a1a0 2024-05-08T00:48:26.763 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:26.769+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:48:26.766 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:26.772+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:48:26.766 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:26.772+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:48:26.766 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:26.772+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:48:29.037 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:29.039+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:48:29.037 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:29.039+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:48:29.037 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:29.039+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:48:29.037 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:29.041+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:48:29.037 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:29.041+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:48:29.037 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:29.041+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:48:29.038 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:29.041+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:48:29.038 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:29.043+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:48:29.038 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:29.043+0000 7f5317d5e640 -1 reset not still connected to 0x5620bdda1110 2024-05-08T00:48:31.360 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:31.366+0000 7fa00a173640 -1 reset not still connected to 0x5564ecdc96c0 2024-05-08T00:48:31.360 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:31.366+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270d00 2024-05-08T00:48:31.360 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:31.366+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270f70 2024-05-08T00:48:31.362 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:48:31.368+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271040 2024-05-08T00:48:31.518 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:31.524+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:48:31.518 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:31.524+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:48:32.610 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.614+0000 7fbb0a622640 -1 reset not still connected to 0x55a78c115ad0 2024-05-08T00:48:32.610 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.616+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:48:32.610 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.616+0000 7fbb0a622640 -1 reset not still connected to 0x55a79b683520 2024-05-08T00:48:32.615 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.621+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a0035d40 2024-05-08T00:48:32.619 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.624+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06229c0 2024-05-08T00:48:32.619 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.624+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06c5110 2024-05-08T00:48:32.619 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.624+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a65c52b0 2024-05-08T00:48:32.619 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.624+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:48:32.619 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.624+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:48:32.619 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.624+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:48:32.619 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.624+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:48:32.619 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:32.624+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:48:33.140 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.142+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:48:33.146 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.152+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:48:33.146 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.152+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:48:33.146 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.152+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:48:33.146 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.152+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:48:33.146 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.152+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:48:33.186 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.192+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:48:33.193 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.197+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06229c0 2024-05-08T00:48:33.193 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.198+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06c5110 2024-05-08T00:48:33.193 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.198+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a65c52b0 2024-05-08T00:48:33.193 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.198+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:48:33.193 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.198+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:48:33.193 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.198+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:48:33.193 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.198+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:48:33.193 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:33.198+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:48:34.541 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:34.547+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:48:34.542 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:34.547+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:48:34.542 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:34.547+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:48:34.542 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:34.547+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:48:39.187 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:48:39.193+0000 7fa2e2566640 -1 bdev(0x5640ce37ee00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 5 2024-05-08T00:48:39.533 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.539+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:48:39.535 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.541+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:48:39.535 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.541+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:48:39.537 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.543+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:48:39.540 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.546+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:48:39.645 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.648+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a270 2024-05-08T00:48:39.645 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.648+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:48:39.645 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.648+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:48:39.645 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.648+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:48:39.647 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.653+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:48:39.647 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.653+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:48:39.650 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.655+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:48:39.650 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.656+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:48:39.650 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.656+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:48:39.650 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.656+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:48:39.650 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.656+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:48:39.650 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.656+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:48:39.652 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.658+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:48:39.652 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.658+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:48:39.652 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.658+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b7f7d1e0 2024-05-08T00:48:39.652 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.658+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:48:39.652 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.658+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:48:39.938 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.943+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:48:39.938 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.943+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06229c0 2024-05-08T00:48:39.941 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.947+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a65c52b0 2024-05-08T00:48:39.948 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.947+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:48:39.948 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.947+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:48:39.948 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.947+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:48:39.948 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.947+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:48:39.948 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.947+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:48:39.948 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.951+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f4a04e0 2024-05-08T00:48:39.948 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.951+0000 7fbb0a622640 -1 reset not still connected to 0x55a77f7c0000 2024-05-08T00:48:39.948 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.951+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:48:39.948 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.951+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:48:39.957 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.960+0000 7fbb0a622640 -1 reset not still connected to 0x55a7960fc0d0 2024-05-08T00:48:39.958 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.965+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06c5110 2024-05-08T00:48:39.958 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.965+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:48:39.959 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.965+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:48:39.966 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.972+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:48:39.966 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:39.972+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:48:40.383 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:40.389+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06229c0 2024-05-08T00:48:40.387 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:40.394+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a65c52b0 2024-05-08T00:48:40.389 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:40.394+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:48:40.390 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:40.395+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:48:40.390 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:40.395+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:48:40.390 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:40.395+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:48:40.390 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:40.395+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:48:42.193 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:42.199+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:48:42.193 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:42.199+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:48:42.197 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:42.203+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:48:42.197 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:42.203+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:48:43.651 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:43.656+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:48:43.652 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:43.656+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:48:43.652 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:43.656+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:48:43.653 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:43.657+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:48:44.200 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.206+0000 7f2ee8664640 -1 reset not still connected to 0x563229ab1ad0 2024-05-08T00:48:44.208 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:48:44.208 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x563235842d00 2024-05-08T00:48:44.209 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:48:44.209 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x56323cfb95f0 2024-05-08T00:48:44.209 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:48:44.209 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x56323d171d40 2024-05-08T00:48:44.209 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x563243799ad0 2024-05-08T00:48:44.209 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd6680 2024-05-08T00:48:44.210 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd68f0 2024-05-08T00:48:44.211 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.214+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:48:44.211 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.215+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0ba00 2024-05-08T00:48:44.211 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.215+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:48:44.211 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.215+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:48:44.211 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:44.215+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:48:45.478 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.483+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:48:45.478 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.483+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:48:45.478 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.483+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:48:45.479 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.485+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:48:45.479 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.485+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:48:45.630 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.634+0000 7f5317d5e640 -1 reset not still connected to 0x562091a9ad00 2024-05-08T00:48:45.631 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.638+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:48:45.641 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.646+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:48:45.641 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.646+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:48:45.641 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.646+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:48:45.641 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.646+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:48:45.641 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.646+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:48:45.641 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.646+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:48:45.641 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.648+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:48:45.642 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.648+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:48:45.642 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:45.648+0000 7f5317d5e640 -1 reset not still connected to 0x5620bdda1110 2024-05-08T00:48:49.238 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.245+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:48:49.249 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.252+0000 7f5317d5e640 -1 reset not still connected to 0x56209393a820 2024-05-08T00:48:49.249 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.252+0000 7f5317d5e640 -1 reset not still connected to 0x562099e6ad00 2024-05-08T00:48:49.251 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.254+0000 7f5317d5e640 -1 reset not still connected to 0x56209ddcc680 2024-05-08T00:48:49.251 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.254+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:48:49.251 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.254+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:48:49.251 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.254+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:48:49.252 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.254+0000 7f5317d5e640 -1 reset not still connected to 0x5620a57cc750 2024-05-08T00:48:49.252 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.254+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:48:49.252 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.256+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:48:49.252 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.256+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:48:49.252 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.256+0000 7f5317d5e640 -1 reset not still connected to 0x5620b325cc30 2024-05-08T00:48:49.252 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.256+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:48:49.252 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.256+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:48:49.252 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.256+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:48:49.253 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:49.256+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:48:50.466 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.471+0000 7f5317d5e640 -1 reset not still connected to 0x56208b731a00 2024-05-08T00:48:50.466 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.472+0000 7f5317d5e640 -1 reset not still connected to 0x56208cf93110 2024-05-08T00:48:50.466 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.472+0000 7f5317d5e640 -1 reset not still connected to 0x562091a9ad00 2024-05-08T00:48:50.466 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.472+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:48:50.467 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.472+0000 7f5317d5e640 -1 reset not still connected to 0x5620961c8000 2024-05-08T00:48:50.469 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x56209736a000 2024-05-08T00:48:50.469 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x562099d02000 2024-05-08T00:48:50.469 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:48:50.469 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620a08b21a0 2024-05-08T00:48:50.469 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:48:50.470 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:48:50.470 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:48:50.470 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:48:50.470 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:48:50.470 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:48:50.470 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:48:50.470 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:48:50.470 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:50.474+0000 7f5317d5e640 -1 reset not still connected to 0x5620bdda1110 2024-05-08T00:48:50.998 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:51.002+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:48:51.001 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:51.005+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:48:51.001 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:51.005+0000 7f2ee8664640 -1 reset not still connected to 0x56323cfb95f0 2024-05-08T00:48:51.001 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:51.005+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:48:51.001 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:51.005+0000 7f2ee8664640 -1 reset not still connected to 0x56323d171d40 2024-05-08T00:48:51.001 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:51.005+0000 7f2ee8664640 -1 reset not still connected to 0x563243799ad0 2024-05-08T00:48:51.001 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:51.005+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd68f0 2024-05-08T00:48:51.001 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:48:51.005+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:48:54.005 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.009+0000 7f5317d5e640 -1 reset not still connected to 0x562092abe8f0 2024-05-08T00:48:54.006 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.011+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:48:54.010 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.015+0000 7f5317d5e640 -1 reset not still connected to 0x562099e6ad00 2024-05-08T00:48:54.010 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.015+0000 7f5317d5e640 -1 reset not still connected to 0x56209ddcc680 2024-05-08T00:48:54.010 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.015+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:48:54.010 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.015+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:48:54.012 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.018+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:48:54.018 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620a57cc750 2024-05-08T00:48:54.018 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:48:54.018 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:48:54.018 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:48:54.018 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620b325cc30 2024-05-08T00:48:54.018 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:48:54.018 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:48:54.018 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.023+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:48:54.020 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.026+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:48:54.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.665+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:48:54.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.665+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:48:54.668 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.674+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:48:54.668 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.674+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:48:54.668 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.674+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:48:54.668 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:48:54.674+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:48:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-05-08T00:48:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep 'image health:' 2024-05-08T00:48:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut '-d ' -f 3 2024-05-08T00:48:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: failed to query services: (13) Permission denied 2024-05-08T00:48:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=OK 2024-05-08T00:48:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test OK = ERROR 2024-05-08T00:48:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test OK = OK 2024-05-08T00:48:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-05-08T00:48:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:48:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_count=50 2024-05-08T00:48:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:48:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local count 2024-05-08T00:48:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_count=0 2024-05-08T00:48:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:48:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 40 2024-05-08T00:48:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in `seq 1 40` 2024-05-08T00:48:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 1 -ne 1 2024-05-08T00:48:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-05-08T00:48:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep 'images: ' 2024-05-08T00:48:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut '-d ' -f 2 2024-05-08T00:48:55.934 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:55.939+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:48:55.937 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:55.941+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:48:55.938 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:55.944+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:48:55.940 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:55.946+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:48:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: failed to query services: (13) Permission denied 2024-05-08T00:48:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ count=50 2024-05-08T00:48:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 50 = 50 2024-05-08T00:48:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-05-08T00:48:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:48:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state 2024-05-08T00:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 40 2024-05-08T00:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in `seq 1 40` 2024-05-08T00:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 1 -ne 1 2024-05-08T00:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-05-08T00:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep 'image health:' 2024-05-08T00:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut '-d ' -f 3 2024-05-08T00:48:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: failed to query services: (13) Permission denied 2024-05-08T00:48:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=OK 2024-05-08T00:48:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test OK = ERROR 2024-05-08T00:48:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test OK = OK 2024-05-08T00:48:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ testlog 'TEST: compare many images' 2024-05-08T00:48:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ tee -a /tmp/tmp.W1VfRby4SU/rbd-mirror.test.log 2024-05-08T00:48:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ date '+%F %T' 2024-05-08T00:48:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo 2024-05-08 00:48:56 TEST: compare many images 2024-05-08T00:48:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:2024-05-08 00:48:56 TEST: compare many images 2024-05-08T00:48:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 50 2024-05-08T00:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_1 2024-05-08T00:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_1 snap 2024-05-08T00:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:48:57.117 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:57.121+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:48:57.119 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:57.125+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:48:57.119 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:48:57.125+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:48:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-05-08T00:48:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:48:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-05-08T00:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:48:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:48:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:48:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:48:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:48:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:48:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:48:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-05-08T00:48:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:no valid command found; 5 closest matches: 2024-05-08T00:48:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_11 2024-05-08T00:48:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_13 2024-05-08T00:48:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_15 2024-05-08T00:48:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_17 2024-05-08T00:48:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_19 2024-05-08T00:48:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:48:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:48:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:48:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:48:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:48:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:48:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:48:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:48:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:48:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:48:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:48:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:48:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:48:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:48:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:48:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-05-08T00:48:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:48:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:48:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:48:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:48:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:48:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:48:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:48:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-05-08T00:48:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:48:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:48:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:48:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_1 2024-05-08T00:48:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:48:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:48:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:48:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:48:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:48:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:48:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:48:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:48:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_1 2024-05-08T00:48:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:48:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:48:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_1 ']' 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:48:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:48:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:48:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:48:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:48:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:48:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:48:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:48:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:48:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:48:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:48:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:48:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:48:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-05-08T00:48:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:no valid command found; 5 closest matches: 2024-05-08T00:48:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_11 2024-05-08T00:48:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_13 2024-05-08T00:48:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_15 2024-05-08T00:48:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_17 2024-05-08T00:48:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_19 2024-05-08T00:48:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:48:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:48:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-05-08T00:48:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:48:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:48:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:48:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:48:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:48:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:48:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:48:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:48:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:48:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:48:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:48:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:48:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-05-08T00:48:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-05-08T00:48:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:48:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:48:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:48:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-05-08T00:48:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:48:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:48:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:48:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:48:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-05-08T00:48:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_1.status 2024-05-08T00:48:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:48:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.status 2024-05-08T00:48:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-05-08T00:48:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.status 2024-05-08T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:48:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12659], [object_number=14, tag_tid=1, entry_tid=12658]]], state=connected] 2024-05-08T00:48:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12659], [object_number=14, tag_tid=1, entry_tid=12658]]], state=connected] 2024-05-08T00:48:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.status 2024-05-08T00:48:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:48:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_1.status 2024-05-08T00:48:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:48:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.status 2024-05-08T00:48:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-05-08T00:48:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.status 2024-05-08T00:48:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:48:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:48:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:48:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12659], [object_number=14, tag_tid=1, entry_tid=12658]]], state=connected] 2024-05-08T00:48:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12659], [object_number=14, tag_tid=1, entry_tid=12658]]], state=connected] 2024-05-08T00:48:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.status 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:48:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:00.152 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.158+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:49:00.152 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.158+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:49:00.153 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.158+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:49:00.153 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.158+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:49:00.153 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.158+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:49:00.157 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.161+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:49:00.158 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.161+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:49:00.158 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.164+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:49:00.160 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:49:00.160 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:49:00.160 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:49:00.161 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a103f6c0 2024-05-08T00:49:00.161 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:49:00.161 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:49:00.161 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:49:00.161 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:49:00.161 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b7f7d1e0 2024-05-08T00:49:00.161 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:49:00.161 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:00.165+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:49:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-05-08T00:49:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_1': 2024-05-08T00:49:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad3f8f4b3e 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad3f8f4b3e 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:40:59 2024 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:40:59 2024 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:40:59 2024 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad3f8f4b3e 2024-05-08T00:49:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 6d1a00ec-f14d-4828-83db-a9519dd152f5 2024-05-08T00:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_1 snap 2024-05-08T00:49:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:49:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.export 2024-05-08T00:49:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_1.export 2024-05-08T00:49:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_1.export 2024-05-08T00:49:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.export 2024-05-08T00:49:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_1.export 2024-05-08T00:49:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_1.export 2024-05-08T00:49:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_1.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_1.export 2024-05-08T00:49:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_2 2024-05-08T00:49:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_2 snap 2024-05-08T00:49:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:49:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:01.650 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.654+0000 7f5317d5e640 -1 reset not still connected to 0x562091a9ad00 2024-05-08T00:49:01.652 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.657+0000 7f5317d5e640 -1 reset not still connected to 0x5620961c8000 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x562099d02000 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x56209d36fd40 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620a08b21a0 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:49:01.661 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:49:01.662 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:49:01.662 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:01.667+0000 7f5317d5e640 -1 reset not still connected to 0x5620bdda1110 2024-05-08T00:49:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-05-08T00:49:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:49:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-05-08T00:49:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:49:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-05-08T00:49:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:49:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-05-08T00:49:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-05-08T00:49:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-05-08T00:49:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_2 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_2 2024-05-08T00:49:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_2 ']' 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-05-08T00:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-05-08T00:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:49:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-05-08T00:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-05-08T00:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_2.status 2024-05-08T00:49:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.status 2024-05-08T00:49:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-05-08T00:49:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.status 2024-05-08T00:49:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9655], [object_number=10, tag_tid=1, entry_tid=9654]]], state=connected] 2024-05-08T00:49:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9655], [object_number=10, tag_tid=1, entry_tid=9654]]], state=connected] 2024-05-08T00:49:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.status 2024-05-08T00:49:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-05-08T00:49:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_2.status 2024-05-08T00:49:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.status 2024-05-08T00:49:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-05-08T00:49:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.status 2024-05-08T00:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9655], [object_number=10, tag_tid=1, entry_tid=9654]]], state=connected] 2024-05-08T00:49:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9655], [object_number=10, tag_tid=1, entry_tid=9654]]], state=connected] 2024-05-08T00:49:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.status 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:05.468 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:05.474+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06229c0 2024-05-08T00:49:05.468 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:05.474+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a06c5110 2024-05-08T00:49:05.468 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:05.474+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b427f5f0 2024-05-08T00:49:05.468 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:05.474+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b81e01a0 2024-05-08T00:49:05.469 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:05.476+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b9976000 2024-05-08T00:49:05.469 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:05.476+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0497c70 2024-05-08T00:49:05.471 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:05.476+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0a184e0 2024-05-08T00:49:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-05-08T00:49:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_2': 2024-05-08T00:49:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa2fbf84d3 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa2fbf84d3 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:41:10 2024 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:41:10 2024 2024-05-08T00:49:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:41:10 2024 2024-05-08T00:49:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa2fbf84d3 2024-05-08T00:49:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: d54fcfd3-9850-428a-8333-0dfb8e192705 2024-05-08T00:49:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_2 snap 2024-05-08T00:49:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:49:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.export 2024-05-08T00:49:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_2.export 2024-05-08T00:49:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_2.export 2024-05-08T00:49:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.export 2024-05-08T00:49:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_2.export 2024-05-08T00:49:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_2.export 2024-05-08T00:49:06.362 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.368+0000 7f2ee8664640 -1 reset not still connected to 0x56320ec0ba00 2024-05-08T00:49:06.365 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.372+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:49:06.366 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.372+0000 7f2ee8664640 -1 reset not still connected to 0x5632159288f0 2024-05-08T00:49:06.366 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.372+0000 7f2ee8664640 -1 reset not still connected to 0x56321c360c30 2024-05-08T00:49:06.366 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.372+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:49:06.366 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.372+0000 7f2ee8664640 -1 reset not still connected to 0x563226151a00 2024-05-08T00:49:06.366 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.372+0000 7f2ee8664640 -1 reset not still connected to 0x563229ab1ad0 2024-05-08T00:49:06.366 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:49:06.366 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x563235842d00 2024-05-08T00:49:06.367 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:49:06.367 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x56323cfb95f0 2024-05-08T00:49:06.367 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:49:06.367 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x56323d171d40 2024-05-08T00:49:06.367 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x563243799ad0 2024-05-08T00:49:06.367 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd6680 2024-05-08T00:49:06.367 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd68f0 2024-05-08T00:49:06.367 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.373+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:49:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_2.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_2.export 2024-05-08T00:49:06.424 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.430+0000 7f2ee8664640 -1 reset not still connected to 0x56320f5cc820 2024-05-08T00:49:06.424 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.430+0000 7f2ee8664640 -1 reset not still connected to 0x5632159288f0 2024-05-08T00:49:06.424 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.430+0000 7f2ee8664640 -1 reset not still connected to 0x56321c360c30 2024-05-08T00:49:06.424 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.430+0000 7f2ee8664640 -1 reset not still connected to 0x563225d8ca90 2024-05-08T00:49:06.424 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.430+0000 7f2ee8664640 -1 reset not still connected to 0x563226151a00 2024-05-08T00:49:06.424 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.430+0000 7f2ee8664640 -1 reset not still connected to 0x563229ab1ad0 2024-05-08T00:49:06.426 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.432+0000 7f2ee8664640 -1 reset not still connected to 0x563232c31860 2024-05-08T00:49:06.426 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.432+0000 7f2ee8664640 -1 reset not still connected to 0x563235842d00 2024-05-08T00:49:06.426 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.432+0000 7f2ee8664640 -1 reset not still connected to 0x563235a69040 2024-05-08T00:49:06.427 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.432+0000 7f2ee8664640 -1 reset not still connected to 0x56323cfb95f0 2024-05-08T00:49:06.427 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.432+0000 7f2ee8664640 -1 reset not still connected to 0x56323d16a750 2024-05-08T00:49:06.427 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.432+0000 7f2ee8664640 -1 reset not still connected to 0x56323d171d40 2024-05-08T00:49:06.427 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.432+0000 7f2ee8664640 -1 reset not still connected to 0x563243799ad0 2024-05-08T00:49:06.427 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.433+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd6680 2024-05-08T00:49:06.427 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.433+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd68f0 2024-05-08T00:49:06.427 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:06.433+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_3 2024-05-08T00:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_3 snap 2024-05-08T00:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:07.240 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.246+0000 7f2ee8664640 -1 reset not still connected to 0x56321c121110 2024-05-08T00:49:07.242 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.250+0000 7f2ee8664640 -1 reset not still connected to 0x56321c336a90 2024-05-08T00:49:07.245 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.250+0000 7f2ee8664640 -1 reset not still connected to 0x56323136a0d0 2024-05-08T00:49:07.245 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.250+0000 7f2ee8664640 -1 reset not still connected to 0x5632319b6f70 2024-05-08T00:49:07.245 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.250+0000 7f2ee8664640 -1 reset not still connected to 0x5632329f7380 2024-05-08T00:49:07.245 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.250+0000 7f2ee8664640 -1 reset not still connected to 0x5632366308f0 2024-05-08T00:49:07.245 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.251+0000 7f2ee8664640 -1 reset not still connected to 0x56323870a1a0 2024-05-08T00:49:07.245 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.251+0000 7f2ee8664640 -1 reset not still connected to 0x563238c0dee0 2024-05-08T00:49:07.245 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.251+0000 7f2ee8664640 -1 reset not still connected to 0x5632456e5e10 2024-05-08T00:49:07.248 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.255+0000 7f2ee8664640 -1 reset not still connected to 0x563245b60b60 2024-05-08T00:49:07.248 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:49:07.255+0000 7f2ee8664640 -1 reset not still connected to 0x56324a909790 2024-05-08T00:49:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-05-08T00:49:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:49:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-05-08T00:49:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-05-08T00:49:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-05-08T00:49:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-05-08T00:49:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_3 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_3 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_3 ']' 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-05-08T00:49:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:49:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-05-08T00:49:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_3.status 2024-05-08T00:49:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.status 2024-05-08T00:49:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-05-08T00:49:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.status 2024-05-08T00:49:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 4 2024-05-08T00:49:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 5 2024-05-08T00:49:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=17427], [object_number=18, tag_tid=1, entry_tid=17426]]], state=connected] 2024-05-08T00:49:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=17427], [object_number=18, tag_tid=1, entry_tid=17426]]], state=connected] 2024-05-08T00:49:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.status 2024-05-08T00:49:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=21, tag_tid=2, entry_tid=1' 2024-05-08T00:49:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-05-08T00:49:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:49:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-05-08T00:49:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:49:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-05-08T00:49:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_3.status 2024-05-08T00:49:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.status 2024-05-08T00:49:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-05-08T00:49:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.status 2024-05-08T00:49:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 4 2024-05-08T00:49:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 5 2024-05-08T00:49:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=17427], [object_number=18, tag_tid=1, entry_tid=17426]]], state=connected] 2024-05-08T00:49:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=1], [object_number=20, tag_tid=2, entry_tid=0], [object_number=19, tag_tid=1, entry_tid=17427], [object_number=18, tag_tid=1, entry_tid=17426]]], state=connected] 2024-05-08T00:49:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.status 2024-05-08T00:49:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=1' 2024-05-08T00:49:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=21, tag_tid=2, entry_tid=1' -a 'object_number=21, tag_tid=2, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=1' 2024-05-08T00:49:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-05-08T00:49:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:49:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:10.532 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.538+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:49:10.534 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.541+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:49:10.534 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.542+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:49:10.543 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.549+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:49:10.544 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.549+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:49:10.544 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.549+0000 7f5317d5e640 -1 reset not still connected to 0x5620b325cc30 2024-05-08T00:49:10.544 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.549+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:49:10.545 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.552+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:49:10.546 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.552+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:49:10.546 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:10.552+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:49:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-05-08T00:49:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_3': 2024-05-08T00:49:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aaf18af999 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aaf18af999 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:41:18 2024 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:41:18 2024 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:42:19 2024 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aaf18af999 2024-05-08T00:49:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 21600a44-edc3-44d2-9824-379c2f4e7f72 2024-05-08T00:49:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_3 snap 2024-05-08T00:49:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:49:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.export 2024-05-08T00:49:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_3.export 2024-05-08T00:49:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_3.export 2024-05-08T00:49:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.export 2024-05-08T00:49:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_3.export 2024-05-08T00:49:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_3.export 2024-05-08T00:49:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_3.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_3.export 2024-05-08T00:49:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_4 2024-05-08T00:49:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_4 snap 2024-05-08T00:49:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:49:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:12.592 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a77fde65b0 2024-05-08T00:49:12.592 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a270 2024-05-08T00:49:12.592 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4a680 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a780b4b450 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d22f70 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d236c0 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ba0 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7819031e0 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a78d826000 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a78e20f860 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962a1110 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7962f9c70 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a00c8f70 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7a103f6c0 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7abeee5b0 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b146cf70 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b301ed00 2024-05-08T00:49:12.593 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b60488f0 2024-05-08T00:49:12.594 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b7f7d1e0 2024-05-08T00:49:12.594 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7b8186000 2024-05-08T00:49:12.594 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:49:12.598+0000 7fbb0a622640 -1 reset not still connected to 0x55a7c0b52b60 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-05-08T00:49:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-05-08T00:49:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:no valid command found; 5 closest matches: 2024-05-08T00:49:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_41 2024-05-08T00:49:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_43 2024-05-08T00:49:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_45 2024-05-08T00:49:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_47 2024-05-08T00:49:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status mirror/image_49 2024-05-08T00:49:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:49:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:49:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-05-08T00:49:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-05-08T00:49:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_4 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_4 2024-05-08T00:49:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_4 ']' 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-05-08T00:49:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-05-08T00:49:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:no valid command found; 5 closest matches: 2024-05-08T00:49:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_41 2024-05-08T00:49:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_43 2024-05-08T00:49:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_45 2024-05-08T00:49:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_47 2024-05-08T00:49:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush mirror/image_49 2024-05-08T00:49:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-05-08T00:49:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:49:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:49:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-05-08T00:49:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-05-08T00:49:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:49:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-05-08T00:49:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-05-08T00:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_4.status 2024-05-08T00:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.status 2024-05-08T00:49:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-05-08T00:49:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.status 2024-05-08T00:49:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:49:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:49:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=10882], [object_number=15, tag_tid=1, entry_tid=10879]]], state=connected] 2024-05-08T00:49:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=10882], [object_number=15, tag_tid=1, entry_tid=10879]]], state=connected] 2024-05-08T00:49:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.status 2024-05-08T00:49:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-05-08T00:49:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:49:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-05-08T00:49:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:49:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-05-08T00:49:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_4.status 2024-05-08T00:49:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.status 2024-05-08T00:49:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-05-08T00:49:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.status 2024-05-08T00:49:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:49:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:49:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=10882], [object_number=15, tag_tid=1, entry_tid=10879]]], state=connected] 2024-05-08T00:49:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=10882], [object_number=15, tag_tid=1, entry_tid=10879]]], state=connected] 2024-05-08T00:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.status 2024-05-08T00:49:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-05-08T00:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-05-08T00:49:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_4': 2024-05-08T00:49:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad8cc41cf 2024-05-08T00:49:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad8cc41cf 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:41:32 2024 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:41:32 2024 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:42:33 2024 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad8cc41cf 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 48f5787c-9c3f-4f5d-bf9e-f93fd077aa83 2024-05-08T00:49:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_4 snap 2024-05-08T00:49:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:49:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.export 2024-05-08T00:49:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_4.export 2024-05-08T00:49:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_4.export 2024-05-08T00:49:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.export 2024-05-08T00:49:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_4.export 2024-05-08T00:49:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_4.export 2024-05-08T00:49:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_4.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_4.export 2024-05-08T00:49:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_5 2024-05-08T00:49:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_5 snap 2024-05-08T00:49:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:49:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:17.357 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:49:17.364+0000 7f9ffbd1a640 -1 bdev(0x5564eb38ce00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 10 2024-05-08T00:49:17.358 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:49:17.365+0000 7f9ffc51b640 -1 bdev(0x5564eb38ce00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 12 2024-05-08T00:49:17.358 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:49:17.366+0000 7f9ff8d14640 -1 bdev(0x5564eb38ce00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 14 2024-05-08T00:49:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-05-08T00:49:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:49:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-05-08T00:49:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:49:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-05-08T00:49:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-05-08T00:49:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-05-08T00:49:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_5 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_5 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_5 ']' 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-05-08T00:49:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-05-08T00:49:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_5.status 2024-05-08T00:49:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.status 2024-05-08T00:49:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-05-08T00:49:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.status 2024-05-08T00:49:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8415], [object_number=10, tag_tid=1, entry_tid=8414]]], state=connected] 2024-05-08T00:49:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8415], [object_number=10, tag_tid=1, entry_tid=8414]]], state=connected] 2024-05-08T00:49:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.status 2024-05-08T00:49:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-05-08T00:49:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:49:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-05-08T00:49:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:49:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-05-08T00:49:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_5.status 2024-05-08T00:49:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.status 2024-05-08T00:49:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-05-08T00:49:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.status 2024-05-08T00:49:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8415], [object_number=10, tag_tid=1, entry_tid=8414]]], state=connected] 2024-05-08T00:49:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8415], [object_number=10, tag_tid=1, entry_tid=8414]]], state=connected] 2024-05-08T00:49:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.status 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-05-08T00:49:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_5': 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa43c61593 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa43c61593 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:41:42 2024 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:41:42 2024 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:41:42 2024 2024-05-08T00:49:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa43c61593 2024-05-08T00:49:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: ae2bd13b-6c00-4cda-98b1-d0e53c5eb3ec 2024-05-08T00:49:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_5 snap 2024-05-08T00:49:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:49:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.export 2024-05-08T00:49:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_5.export 2024-05-08T00:49:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_5.export 2024-05-08T00:49:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.export 2024-05-08T00:49:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_5.export 2024-05-08T00:49:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_5.export 2024-05-08T00:49:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_5.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_5.export 2024-05-08T00:49:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_6 2024-05-08T00:49:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_6 snap 2024-05-08T00:49:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:49:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-05-08T00:49:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:49:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-05-08T00:49:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:49:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-05-08T00:49:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:49:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-05-08T00:49:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-05-08T00:49:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:49:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-05-08T00:49:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:49:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:49:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-05-08T00:49:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-05-08T00:49:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_6 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_6 2024-05-08T00:49:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_6 ']' 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-05-08T00:49:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-05-08T00:49:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:49:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-05-08T00:49:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-05-08T00:49:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:49:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-05-08T00:49:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:49:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-05-08T00:49:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_6.status 2024-05-08T00:49:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.status 2024-05-08T00:49:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-05-08T00:49:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.status 2024-05-08T00:49:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:49:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:49:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10787], [object_number=14, tag_tid=1, entry_tid=10786]]], state=connected] 2024-05-08T00:49:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10787], [object_number=14, tag_tid=1, entry_tid=10786]]], state=connected] 2024-05-08T00:49:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.status 2024-05-08T00:49:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-05-08T00:49:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_6.status 2024-05-08T00:49:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.status 2024-05-08T00:49:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-05-08T00:49:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.status 2024-05-08T00:49:27.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:49:27.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:49:27.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10787], [object_number=14, tag_tid=1, entry_tid=10786]]], state=connected] 2024-05-08T00:49:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=10787], [object_number=14, tag_tid=1, entry_tid=10786]]], state=connected] 2024-05-08T00:49:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.status 2024-05-08T00:49:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-05-08T00:49:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:49:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_6': 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10adf91fa034 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10adf91fa034 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:41:50 2024 2024-05-08T00:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:41:50 2024 2024-05-08T00:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:42:51 2024 2024-05-08T00:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10adf91fa034 2024-05-08T00:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: b7e4d902-9c59-4e35-9af2-eaf8ab4857d3 2024-05-08T00:49:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_6 snap 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.export 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_6.export 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_6.export 2024-05-08T00:49:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.export 2024-05-08T00:49:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_6.export 2024-05-08T00:49:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_6.export 2024-05-08T00:49:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_6.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_6.export 2024-05-08T00:49:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_7 2024-05-08T00:49:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_7 snap 2024-05-08T00:49:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:49:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-05-08T00:49:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:31.195 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.202+0000 7f5317d5e640 -1 reset not still connected to 0x562084709450 2024-05-08T00:49:31.196 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.204+0000 7f5317d5e640 -1 reset not still connected to 0x562085630a90 2024-05-08T00:49:31.196 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.204+0000 7f5317d5e640 -1 reset not still connected to 0x5620856915f0 2024-05-08T00:49:31.196 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.204+0000 7f5317d5e640 -1 reset not still connected to 0x56208a740410 2024-05-08T00:49:31.197 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.204+0000 7f5317d5e640 -1 reset not still connected to 0x56208ad8f5f0 2024-05-08T00:49:31.197 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.204+0000 7f5317d5e640 -1 reset not still connected to 0x56208b731a00 2024-05-08T00:49:31.205 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x56208cf93110 2024-05-08T00:49:31.205 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x562091a9ad00 2024-05-08T00:49:31.205 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x562091efca90 2024-05-08T00:49:31.205 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620961c8000 2024-05-08T00:49:31.205 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x56209736a000 2024-05-08T00:49:31.205 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x562099d02000 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x56209b4b3860 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x56209d36fd40 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620a08b21a0 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620a37c7110 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620a42cf5f0 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620a5f9c000 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620a92c8750 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620b03f5d40 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620b09581a0 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0bbc000 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6da04e0 2024-05-08T00:49:31.206 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:31.213+0000 7f5317d5e640 -1 reset not still connected to 0x5620bdda1110 2024-05-08T00:49:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-05-08T00:49:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:49:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-05-08T00:49:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-05-08T00:49:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:31.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-05-08T00:49:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_7 2024-05-08T00:49:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:49:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_7 2024-05-08T00:49:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:49:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_7 ']' 2024-05-08T00:49:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-05-08T00:49:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-05-08T00:49:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-05-08T00:49:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-05-08T00:49:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-05-08T00:49:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:49:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-05-08T00:49:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:49:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-05-08T00:49:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_7.status 2024-05-08T00:49:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.status 2024-05-08T00:49:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-05-08T00:49:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.status 2024-05-08T00:49:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:49:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:49:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11279], [object_number=14, tag_tid=1, entry_tid=11278]]], state=connected] 2024-05-08T00:49:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11279], [object_number=14, tag_tid=1, entry_tid=11278]]], state=connected] 2024-05-08T00:49:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.status 2024-05-08T00:49:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-05-08T00:49:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:49:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-05-08T00:49:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:49:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-05-08T00:49:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_7.status 2024-05-08T00:49:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.status 2024-05-08T00:49:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-05-08T00:49:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.status 2024-05-08T00:49:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:49:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:49:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11279], [object_number=14, tag_tid=1, entry_tid=11278]]], state=connected] 2024-05-08T00:49:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11279], [object_number=14, tag_tid=1, entry_tid=11278]]], state=connected] 2024-05-08T00:49:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.status 2024-05-08T00:49:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-05-08T00:49:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:49:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-05-08T00:49:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_7': 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa2d1b9dde 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa2d1b9dde 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:42:00 2024 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:42:00 2024 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:43:01 2024 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa2d1b9dde 2024-05-08T00:49:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 517beb73-838f-4363-a18e-30cbbe2394fc 2024-05-08T00:49:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_7 snap 2024-05-08T00:49:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:49:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.export 2024-05-08T00:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_7.export 2024-05-08T00:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_7.export 2024-05-08T00:49:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.export 2024-05-08T00:49:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_7.export 2024-05-08T00:49:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_7.export 2024-05-08T00:49:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_7.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_7.export 2024-05-08T00:49:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_8 2024-05-08T00:49:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_8 snap 2024-05-08T00:49:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:49:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-05-08T00:49:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:49:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-05-08T00:49:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:49:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-05-08T00:49:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-05-08T00:49:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:49:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:49:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:49:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-05-08T00:49:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_8 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:49:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_8 2024-05-08T00:49:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_8 ']' 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-05-08T00:49:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-05-08T00:49:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:49:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-05-08T00:49:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_8.status 2024-05-08T00:49:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.status 2024-05-08T00:49:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-05-08T00:49:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.status 2024-05-08T00:49:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:49:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:49:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12998], [object_number=15, tag_tid=1, entry_tid=12995]]], state=connected] 2024-05-08T00:49:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12998], [object_number=15, tag_tid=1, entry_tid=12995]]], state=connected] 2024-05-08T00:49:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.status 2024-05-08T00:49:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-05-08T00:49:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:49:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-05-08T00:49:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:49:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-05-08T00:49:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_8.status 2024-05-08T00:49:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.status 2024-05-08T00:49:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-05-08T00:49:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.status 2024-05-08T00:49:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:49:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:49:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12998], [object_number=15, tag_tid=1, entry_tid=12995]]], state=connected] 2024-05-08T00:49:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12998], [object_number=15, tag_tid=1, entry_tid=12995]]], state=connected] 2024-05-08T00:49:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.status 2024-05-08T00:49:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:49:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-05-08T00:49:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:49:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-05-08T00:49:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_8': 2024-05-08T00:49:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad9e8dcce9 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad9e8dcce9 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:42:10 2024 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:42:10 2024 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:43:11 2024 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad9e8dcce9 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: ced4e287-a918-419d-ad5f-27881e9212f8 2024-05-08T00:49:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_8 snap 2024-05-08T00:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.export 2024-05-08T00:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_8.export 2024-05-08T00:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_8.export 2024-05-08T00:49:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.export 2024-05-08T00:49:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_8.export 2024-05-08T00:49:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_8.export 2024-05-08T00:49:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_8.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_8.export 2024-05-08T00:49:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_9 2024-05-08T00:49:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_9 snap 2024-05-08T00:49:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:49:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-05-08T00:49:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:49:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-05-08T00:49:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_9 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:49:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_9 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_9 ']' 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_9' 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2024-05-08T00:49:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2024-05-08T00:49:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_9 2024-05-08T00:49:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T00:49:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_9 '' 2024-05-08T00:49:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T00:49:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-05-08T00:49:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_9.status 2024-05-08T00:49:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.status 2024-05-08T00:49:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-05-08T00:49:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.status 2024-05-08T00:49:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:49:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:49:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6815], [object_number=6, tag_tid=1, entry_tid=6814]]], state=connected] 2024-05-08T00:49:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6815], [object_number=6, tag_tid=1, entry_tid=6814]]], state=connected] 2024-05-08T00:49:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.status 2024-05-08T00:49:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_9.status 2024-05-08T00:49:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.status 2024-05-08T00:49:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-05-08T00:49:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.status 2024-05-08T00:49:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:49:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:49:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6815], [object_number=6, tag_tid=1, entry_tid=6814]]], state=connected] 2024-05-08T00:49:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6815], [object_number=6, tag_tid=1, entry_tid=6814]]], state=connected] 2024-05-08T00:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.status 2024-05-08T00:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-05-08T00:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:43.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-05-08T00:49:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_9': 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa474f5fa5 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa474f5fa5 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:42:22 2024 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:42:22 2024 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:42:22 2024 2024-05-08T00:49:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa474f5fa5 2024-05-08T00:49:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 201331dd-70ae-4350-b986-f60371d24689 2024-05-08T00:49:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_9 snap 2024-05-08T00:49:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:49:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.export 2024-05-08T00:49:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_9.export 2024-05-08T00:49:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_9.export 2024-05-08T00:49:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.export 2024-05-08T00:49:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_9.export 2024-05-08T00:49:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_9.export 2024-05-08T00:49:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_9.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_9.export 2024-05-08T00:49:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_10 2024-05-08T00:49:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_10 snap 2024-05-08T00:49:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:49:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:46.358 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:49:46.366+0000 7fa00a173640 -1 reset not still connected to 0x5564ecdc96c0 2024-05-08T00:49:46.359 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:49:46.366+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270d00 2024-05-08T00:49:46.359 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:49:46.366+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270f70 2024-05-08T00:49:46.360 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:49:46.368+0000 7fa00a173640 -1 reset not still connected to 0x5564ee271040 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-05-08T00:49:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-05-08T00:49:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-05-08T00:49:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-05-08T00:49:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:49:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-05-08T00:49:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:49:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:49:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-05-08T00:49:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_10 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:49:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_10 2024-05-08T00:49:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:49:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_10 ']' 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-05-08T00:49:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-05-08T00:49:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:49:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-05-08T00:49:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-05-08T00:49:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T00:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-05-08T00:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T00:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-05-08T00:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_10.status 2024-05-08T00:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.status 2024-05-08T00:49:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-05-08T00:49:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.status 2024-05-08T00:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8878], [object_number=11, tag_tid=1, entry_tid=8875]]], state=connected] 2024-05-08T00:49:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8878], [object_number=11, tag_tid=1, entry_tid=8875]]], state=connected] 2024-05-08T00:49:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.status 2024-05-08T00:49:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-05-08T00:49:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T00:49:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-05-08T00:49:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T00:49:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-05-08T00:49:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_10.status 2024-05-08T00:49:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.status 2024-05-08T00:49:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-05-08T00:49:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.status 2024-05-08T00:49:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8878], [object_number=11, tag_tid=1, entry_tid=8875]]], state=connected] 2024-05-08T00:49:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8878], [object_number=11, tag_tid=1, entry_tid=8875]]], state=connected] 2024-05-08T00:49:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.status 2024-05-08T00:49:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-05-08T00:49:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:49:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-05-08T00:49:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_10': 2024-05-08T00:49:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad7072b691 2024-05-08T00:49:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad7072b691 2024-05-08T00:49:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:42:29 2024 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:42:29 2024 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:43:30 2024 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad7072b691 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 50d85d4e-6bf5-4e7f-a956-d4c493be38a3 2024-05-08T00:49:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_10 snap 2024-05-08T00:49:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:49:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.export 2024-05-08T00:49:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_10.export 2024-05-08T00:49:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_10.export 2024-05-08T00:49:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.export 2024-05-08T00:49:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_10.export 2024-05-08T00:49:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_10.export 2024-05-08T00:49:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_10.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_10.export 2024-05-08T00:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_11 2024-05-08T00:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_11 snap 2024-05-08T00:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-05-08T00:49:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-05-08T00:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-05-08T00:49:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-05-08T00:49:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_11 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_11 2024-05-08T00:49:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_11 ']' 2024-05-08T00:49:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-05-08T00:49:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-05-08T00:49:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-05-08T00:49:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-05-08T00:49:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_11.status 2024-05-08T00:49:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-05-08T00:49:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:49:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:49:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_11.status 2024-05-08T00:49:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-05-08T00:49:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:49:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:49:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-05-08T00:49:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:49:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-05-08T00:49:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:49:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:49:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:49:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=3 2024-05-08T00:49:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:49:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:49:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=2 2024-05-08T00:49:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:49:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:49:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:49:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:49:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:49:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:49:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=1 2024-05-08T00:49:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 3 = 2 -a 1 -le 1 2024-05-08T00:49:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_11 2024-05-08T00:49:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_11 ']' 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-05-08T00:49:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-05-08T00:49:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_11.status 2024-05-08T00:49:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-05-08T00:49:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:49:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:49:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:49:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-05-08T00:49:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-05-08T00:49:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-05-08T00:49:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_11.status 2024-05-08T00:49:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-05-08T00:49:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:49:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:49:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:49:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.4 2024-05-08T00:49:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_11 2024-05-08T00:49:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_11 ']' 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-05-08T00:49:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-05-08T00:49:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_11.status 2024-05-08T00:49:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-05-08T00:49:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:49:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:49:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:49:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-05-08T00:49:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-05-08T00:49:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:49:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-05-08T00:49:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_11.status 2024-05-08T00:49:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-05-08T00:49:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:49:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:49:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6158], [object_number=7, tag_tid=1, entry_tid=6155]]], state=connected] 2024-05-08T00:49:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.status 2024-05-08T00:49:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:49:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:49:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-05-08T00:49:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-05-08T00:49:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_11': 2024-05-08T00:49:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:49:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:49:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa8271149a 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa8271149a 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:42:38 2024 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:42:38 2024 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:43:47 2024 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa8271149a 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:49:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 43644b2c-476c-418e-8c36-c57083c00dd8 2024-05-08T00:49:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:49:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_11 snap 2024-05-08T00:49:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:49:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:49:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.export 2024-05-08T00:49:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_11.export 2024-05-08T00:49:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_11.export 2024-05-08T00:49:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.export 2024-05-08T00:49:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_11.export 2024-05-08T00:49:54.316 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:49:54.323+0000 7f5307901640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-05-08T00:49:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:49:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_11.export 2024-05-08T00:49:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_11.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_11.export 2024-05-08T00:49:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:49:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_12 2024-05-08T00:49:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_12 snap 2024-05-08T00:49:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:49:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:49:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:49:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:49:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:49:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:49:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:49:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-05-08T00:49:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:49:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:49:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-05-08T00:49:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-05-08T00:49:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:49:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-05-08T00:49:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:49:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:49:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:49:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:49:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-05-08T00:49:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:49:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:49:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:49:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:49:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_12 2024-05-08T00:49:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:49:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:49:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:49:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:49:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:49:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_12 2024-05-08T00:49:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:49:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_12 ']' 2024-05-08T00:49:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-05-08T00:49:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-05-08T00:49:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-05-08T00:49:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:49:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-05-08T00:49:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:49:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-05-08T00:49:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-05-08T00:49:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_12.status 2024-05-08T00:49:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-05-08T00:49:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9975], [object_number=10, tag_tid=1, entry_tid=9974]]], state=connected] 2024-05-08T00:49:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9977], [object_number=8, tag_tid=1, entry_tid=9976], [object_number=11, tag_tid=1, entry_tid=9975], [object_number=10, tag_tid=1, entry_tid=9974]]], state=connected] 2024-05-08T00:49:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_12.status 2024-05-08T00:49:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:58.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:58.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-05-08T00:49:58.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9975], [object_number=10, tag_tid=1, entry_tid=9974]]], state=connected] 2024-05-08T00:49:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=1, entry_tid=9977], [object_number=8, tag_tid=1, entry_tid=9976], [object_number=11, tag_tid=1, entry_tid=9975], [object_number=10, tag_tid=1, entry_tid=9974]]], state=connected] 2024-05-08T00:49:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=1, entry_tid=9977' 2024-05-08T00:49:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=1, entry_tid=9977' 2024-05-08T00:49:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9977' '!=' '' 2024-05-08T00:49:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:49:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=9, tag_tid=1, entry_tid=9977' = '' 2024-05-08T00:49:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=9, tag_tid=1, entry_tid=9977' 2024-05-08T00:49:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:49:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:49:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=2 2024-05-08T00:49:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9977' 2024-05-08T00:49:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:49:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:49:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=1 2024-05-08T00:49:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:49:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:49:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:49:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=9, tag_tid=1, entry_tid=9977' 2024-05-08T00:49:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:49:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:49:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=9977 2024-05-08T00:49:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 2 = 1 -a 1 -le 9977 2024-05-08T00:49:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:49:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:49:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:49:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_12 2024-05-08T00:49:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_12 ']' 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:49:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:49:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:49:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-05-08T00:49:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:49:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:49:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:49:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:49:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:49:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:49:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:49:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-05-08T00:49:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-05-08T00:49:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:49:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-05-08T00:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:49:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:49:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:49:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-05-08T00:49:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-05-08T00:49:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:49:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-05-08T00:49:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-05-08T00:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_12.status 2024-05-08T00:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-05-08T00:49:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9975], [object_number=10, tag_tid=1, entry_tid=9974]]], state=connected] 2024-05-08T00:49:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9975], [object_number=10, tag_tid=1, entry_tid=9974]]], state=connected] 2024-05-08T00:49:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_12.status 2024-05-08T00:49:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:49:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-05-08T00:49:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:49:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:49:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:49:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9975], [object_number=10, tag_tid=1, entry_tid=9974]]], state=connected] 2024-05-08T00:49:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9975], [object_number=10, tag_tid=1, entry_tid=9974]]], state=connected] 2024-05-08T00:49:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.status 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:49:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:49:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_12': 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10adbcf5339a 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10adbcf5339a 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:42:44 2024 2024-05-08T00:50:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:42:44 2024 2024-05-08T00:50:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:43:45 2024 2024-05-08T00:50:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10adbcf5339a 2024-05-08T00:50:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: b84085a5-99cc-4329-a87a-4731ab042bc6 2024-05-08T00:50:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_12 snap 2024-05-08T00:50:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:50:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.export 2024-05-08T00:50:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_12.export 2024-05-08T00:50:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_12.export 2024-05-08T00:50:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.export 2024-05-08T00:50:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_12.export 2024-05-08T00:50:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_12.export 2024-05-08T00:50:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_12.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_12.export 2024-05-08T00:50:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_13 2024-05-08T00:50:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_13 snap 2024-05-08T00:50:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-05-08T00:50:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-05-08T00:50:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-05-08T00:50:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-05-08T00:50:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-05-08T00:50:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_13 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_13 2024-05-08T00:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_13 ']' 2024-05-08T00:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-05-08T00:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-05-08T00:50:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-05-08T00:50:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-05-08T00:50:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_13.status 2024-05-08T00:50:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-05-08T00:50:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11263], [object_number=14, tag_tid=1, entry_tid=11262]]], state=connected] 2024-05-08T00:50:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=11264], [object_number=15, tag_tid=1, entry_tid=11263], [object_number=14, tag_tid=1, entry_tid=11262], [object_number=13, tag_tid=1, entry_tid=11261]]], state=connected] 2024-05-08T00:50:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_13.status 2024-05-08T00:50:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-05-08T00:50:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11263], [object_number=14, tag_tid=1, entry_tid=11262]]], state=connected] 2024-05-08T00:50:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=12, tag_tid=1, entry_tid=11264], [object_number=15, tag_tid=1, entry_tid=11263], [object_number=14, tag_tid=1, entry_tid=11262], [object_number=13, tag_tid=1, entry_tid=11261]]], state=connected] 2024-05-08T00:50:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=12, tag_tid=1, entry_tid=11264' 2024-05-08T00:50:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=12, tag_tid=1, entry_tid=11264' 2024-05-08T00:50:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=11264' '!=' '' 2024-05-08T00:50:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:50:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=12, tag_tid=1, entry_tid=11264' = '' 2024-05-08T00:50:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=12, tag_tid=1, entry_tid=11264' 2024-05-08T00:50:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:50:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:50:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=2 2024-05-08T00:50:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=11264' 2024-05-08T00:50:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:50:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:50:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=1 2024-05-08T00:50:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:50:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:50:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:50:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=12, tag_tid=1, entry_tid=11264' 2024-05-08T00:50:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:50:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:50:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=11264 2024-05-08T00:50:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 2 = 1 -a 1 -le 11264 2024-05-08T00:50:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_13 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_13 ']' 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-05-08T00:50:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_13.status 2024-05-08T00:50:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-05-08T00:50:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11263], [object_number=14, tag_tid=1, entry_tid=11262]]], state=connected] 2024-05-08T00:50:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11263], [object_number=14, tag_tid=1, entry_tid=11262]]], state=connected] 2024-05-08T00:50:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:50:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-05-08T00:50:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_13.status 2024-05-08T00:50:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-05-08T00:50:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11263], [object_number=14, tag_tid=1, entry_tid=11262]]], state=connected] 2024-05-08T00:50:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11263], [object_number=14, tag_tid=1, entry_tid=11262]]], state=connected] 2024-05-08T00:50:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.status 2024-05-08T00:50:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-05-08T00:50:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-05-08T00:50:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_13': 2024-05-08T00:50:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aadf2081f 2024-05-08T00:50:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aadf2081f 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:42:54 2024 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:42:54 2024 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:43:55 2024 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aadf2081f 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: df6609c9-5cb1-41c7-aa59-ba03d7951449 2024-05-08T00:50:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_13 snap 2024-05-08T00:50:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:50:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.export 2024-05-08T00:50:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_13.export 2024-05-08T00:50:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_13.export 2024-05-08T00:50:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.export 2024-05-08T00:50:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_13.export 2024-05-08T00:50:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_13.export 2024-05-08T00:50:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_13.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_13.export 2024-05-08T00:50:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_14 2024-05-08T00:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_14 snap 2024-05-08T00:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-05-08T00:50:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-05-08T00:50:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-05-08T00:50:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:50:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:50:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:50:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-05-08T00:50:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-05-08T00:50:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_14 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_14 2024-05-08T00:50:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_14 ']' 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-05-08T00:50:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-05-08T00:50:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:50:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:10.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:10.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-05-08T00:50:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-05-08T00:50:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_14 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_14 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_14.status 2024-05-08T00:50:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.status 2024-05-08T00:50:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-05-08T00:50:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.status 2024-05-08T00:50:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12026], [object_number=15, tag_tid=1, entry_tid=12023]]], state=connected] 2024-05-08T00:50:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12026], [object_number=15, tag_tid=1, entry_tid=12023]]], state=connected] 2024-05-08T00:50:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.status 2024-05-08T00:50:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-05-08T00:50:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_14 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_14 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_14.status 2024-05-08T00:50:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.status 2024-05-08T00:50:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-05-08T00:50:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.status 2024-05-08T00:50:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12026], [object_number=15, tag_tid=1, entry_tid=12023]]], state=connected] 2024-05-08T00:50:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12026], [object_number=15, tag_tid=1, entry_tid=12023]]], state=connected] 2024-05-08T00:50:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.status 2024-05-08T00:50:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-05-08T00:50:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_14': 2024-05-08T00:50:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ade2304bbd 2024-05-08T00:50:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ade2304bbd 2024-05-08T00:50:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:43:05 2024 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:43:05 2024 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:44:06 2024 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ade2304bbd 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 35acf80c-873c-4bf4-bb5b-0f155393e227 2024-05-08T00:50:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_14 snap 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.export 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_14.export 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_14.export 2024-05-08T00:50:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.export 2024-05-08T00:50:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_14.export 2024-05-08T00:50:11.747 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:50:11.756+0000 7f5307901640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-05-08T00:50:11.748 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:50:11.756+0000 7f5308903640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-05-08T00:50:11.748 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:50:11.757+0000 7f53060fe640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-05-08T00:50:11.752 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:50:11.758+0000 7f53050fc640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 7 2024-05-08T00:50:11.752 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:50:11.758+0000 7f53040fa640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-05-08T00:50:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_14.export 2024-05-08T00:50:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_14.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_14.export 2024-05-08T00:50:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_15 2024-05-08T00:50:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_15 snap 2024-05-08T00:50:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:50:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-05-08T00:50:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:50:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-05-08T00:50:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:50:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-05-08T00:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-05-08T00:50:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_15 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:50:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_15 2024-05-08T00:50:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:50:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:14.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_15 ']' 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-05-08T00:50:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-05-08T00:50:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_15 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_15 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_15.status 2024-05-08T00:50:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.status 2024-05-08T00:50:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-05-08T00:50:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.status 2024-05-08T00:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5559], [object_number=6, tag_tid=1, entry_tid=5558]]], state=connected] 2024-05-08T00:50:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5559], [object_number=6, tag_tid=1, entry_tid=5558]]], state=connected] 2024-05-08T00:50:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.status 2024-05-08T00:50:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:50:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-05-08T00:50:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_15 2024-05-08T00:50:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-05-08T00:50:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_15 2024-05-08T00:50:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-05-08T00:50:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_15.status 2024-05-08T00:50:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.status 2024-05-08T00:50:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-05-08T00:50:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.status 2024-05-08T00:50:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:50:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:50:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5559], [object_number=6, tag_tid=1, entry_tid=5558]]], state=connected] 2024-05-08T00:50:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5559], [object_number=6, tag_tid=1, entry_tid=5558]]], state=connected] 2024-05-08T00:50:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.status 2024-05-08T00:50:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:50:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:50:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-05-08T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-05-08T00:50:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_15': 2024-05-08T00:50:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aaff60863 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aaff60863 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:43:17 2024 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:43:17 2024 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:43:17 2024 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aaff60863 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: f82e7768-10e5-40a5-bbf1-25a7a415412f 2024-05-08T00:50:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_15 snap 2024-05-08T00:50:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:50:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.export 2024-05-08T00:50:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_15.export 2024-05-08T00:50:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_15.export 2024-05-08T00:50:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.export 2024-05-08T00:50:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_15.export 2024-05-08T00:50:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_15.export 2024-05-08T00:50:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_15.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_15.export 2024-05-08T00:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_16 2024-05-08T00:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_16 snap 2024-05-08T00:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-05-08T00:50:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:50:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-05-08T00:50:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:50:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-05-08T00:50:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-05-08T00:50:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-05-08T00:50:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:50:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:50:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:50:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-05-08T00:50:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_16 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:50:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_16 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_16 ']' 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-05-08T00:50:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-05-08T00:50:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:50:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:50:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-05-08T00:50:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_16 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_16 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_16.status 2024-05-08T00:50:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.status 2024-05-08T00:50:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-05-08T00:50:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.status 2024-05-08T00:50:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8799], [object_number=10, tag_tid=1, entry_tid=8798]]], state=connected] 2024-05-08T00:50:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8799], [object_number=10, tag_tid=1, entry_tid=8798]]], state=connected] 2024-05-08T00:50:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.status 2024-05-08T00:50:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-05-08T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_16 2024-05-08T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-05-08T00:50:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_16 2024-05-08T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-05-08T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_16.status 2024-05-08T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.status 2024-05-08T00:50:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-05-08T00:50:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.status 2024-05-08T00:50:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8799], [object_number=10, tag_tid=1, entry_tid=8798]]], state=connected] 2024-05-08T00:50:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8799], [object_number=10, tag_tid=1, entry_tid=8798]]], state=connected] 2024-05-08T00:50:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.status 2024-05-08T00:50:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-05-08T00:50:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:50:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-05-08T00:50:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_16': 2024-05-08T00:50:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10adf7eb6fb0 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10adf7eb6fb0 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:43:23 2024 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:43:23 2024 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:44:24 2024 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10adf7eb6fb0 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: bd214fc9-e1da-41c1-b708-d3aff7b38b2f 2024-05-08T00:50:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_16 snap 2024-05-08T00:50:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:50:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.export 2024-05-08T00:50:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_16.export 2024-05-08T00:50:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_16.export 2024-05-08T00:50:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.export 2024-05-08T00:50:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_16.export 2024-05-08T00:50:21.402 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:50:21.411+0000 7f53038f9640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-05-08T00:50:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_16.export 2024-05-08T00:50:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_16.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_16.export 2024-05-08T00:50:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_17 2024-05-08T00:50:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_17 snap 2024-05-08T00:50:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:50:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-05-08T00:50:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:50:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-05-08T00:50:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:50:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-05-08T00:50:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:50:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-05-08T00:50:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-05-08T00:50:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_17 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_17 2024-05-08T00:50:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:50:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_17 ']' 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-05-08T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-05-08T00:50:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-05-08T00:50:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_17 2024-05-08T00:50:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-05-08T00:50:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_17 2024-05-08T00:50:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-05-08T00:50:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_17.status 2024-05-08T00:50:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.status 2024-05-08T00:50:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-05-08T00:50:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.status 2024-05-08T00:50:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9051], [object_number=10, tag_tid=1, entry_tid=9050]]], state=connected] 2024-05-08T00:50:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9051], [object_number=10, tag_tid=1, entry_tid=9050]]], state=connected] 2024-05-08T00:50:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.status 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_17 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_17 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-05-08T00:50:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_17.status 2024-05-08T00:50:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.status 2024-05-08T00:50:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-05-08T00:50:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.status 2024-05-08T00:50:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9051], [object_number=10, tag_tid=1, entry_tid=9050]]], state=connected] 2024-05-08T00:50:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9051], [object_number=10, tag_tid=1, entry_tid=9050]]], state=connected] 2024-05-08T00:50:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.status 2024-05-08T00:50:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-05-08T00:50:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:50:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-05-08T00:50:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_17': 2024-05-08T00:50:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aacab4dace 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aacab4dace 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:43:33 2024 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:43:33 2024 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:44:34 2024 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aacab4dace 2024-05-08T00:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: d0320aa0-35fa-4a27-b43b-8292bcd483b2 2024-05-08T00:50:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_17 snap 2024-05-08T00:50:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:50:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.export 2024-05-08T00:50:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_17.export 2024-05-08T00:50:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_17.export 2024-05-08T00:50:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.export 2024-05-08T00:50:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_17.export 2024-05-08T00:50:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_17.export 2024-05-08T00:50:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_17.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_17.export 2024-05-08T00:50:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_18 2024-05-08T00:50:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_18 snap 2024-05-08T00:50:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:50:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-05-08T00:50:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:50:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-05-08T00:50:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:50:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-05-08T00:50:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-05-08T00:50:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-05-08T00:50:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:50:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:50:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-05-08T00:50:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_18 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:50:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_18 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_18 ']' 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-05-08T00:50:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-05-08T00:50:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:50:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-05-08T00:50:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:50:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:50:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_18 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_18 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-05-08T00:50:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_18.status 2024-05-08T00:50:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.status 2024-05-08T00:50:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-05-08T00:50:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.status 2024-05-08T00:50:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11679], [object_number=14, tag_tid=1, entry_tid=11678]]], state=connected] 2024-05-08T00:50:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11679], [object_number=14, tag_tid=1, entry_tid=11678]]], state=connected] 2024-05-08T00:50:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.status 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_18 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_18 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_18.status 2024-05-08T00:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.status 2024-05-08T00:50:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-05-08T00:50:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.status 2024-05-08T00:50:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11679], [object_number=14, tag_tid=1, entry_tid=11678]]], state=connected] 2024-05-08T00:50:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11679], [object_number=14, tag_tid=1, entry_tid=11678]]], state=connected] 2024-05-08T00:50:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.status 2024-05-08T00:50:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-05-08T00:50:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:50:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_18': 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad2c0dc82a 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad2c0dc82a 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:43:44 2024 2024-05-08T00:50:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:43:44 2024 2024-05-08T00:50:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:44:45 2024 2024-05-08T00:50:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad2c0dc82a 2024-05-08T00:50:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: b21feb6e-033e-4f31-b7a5-95e91b8b11c1 2024-05-08T00:50:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_18 snap 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.export 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_18.export 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_18.export 2024-05-08T00:50:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.export 2024-05-08T00:50:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_18.export 2024-05-08T00:50:32.716 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:50:32.725+0000 7f5307901640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-05-08T00:50:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_18.export 2024-05-08T00:50:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_18.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_18.export 2024-05-08T00:50:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_19 2024-05-08T00:50:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_19 snap 2024-05-08T00:50:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:50:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-05-08T00:50:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:50:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-05-08T00:50:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:50:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-05-08T00:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-05-08T00:50:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_19 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_19 2024-05-08T00:50:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_19 ']' 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-05-08T00:50:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_19 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_19 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_19.status 2024-05-08T00:50:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.status 2024-05-08T00:50:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-05-08T00:50:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.status 2024-05-08T00:50:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8098], [object_number=11, tag_tid=1, entry_tid=8095]]], state=connected] 2024-05-08T00:50:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8098], [object_number=11, tag_tid=1, entry_tid=8095]]], state=connected] 2024-05-08T00:50:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.status 2024-05-08T00:50:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_19 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_19 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_19.status 2024-05-08T00:50:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.status 2024-05-08T00:50:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-05-08T00:50:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.status 2024-05-08T00:50:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8098], [object_number=11, tag_tid=1, entry_tid=8095]]], state=connected] 2024-05-08T00:50:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8098], [object_number=11, tag_tid=1, entry_tid=8095]]], state=connected] 2024-05-08T00:50:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.status 2024-05-08T00:50:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-05-08T00:50:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:50:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_19': 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa99fc47eb 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa99fc47eb 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:43:58 2024 2024-05-08T00:50:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:43:58 2024 2024-05-08T00:50:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:44:59 2024 2024-05-08T00:50:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa99fc47eb 2024-05-08T00:50:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 89937d44-abf9-4d96-8c37-57e2da3cb6f1 2024-05-08T00:50:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_19 snap 2024-05-08T00:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:50:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.export 2024-05-08T00:50:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_19.export 2024-05-08T00:50:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_19.export 2024-05-08T00:50:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.export 2024-05-08T00:50:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_19.export 2024-05-08T00:50:38.278 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:50:38.286+0000 7f5308102640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 2 2024-05-08T00:50:38.279 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:50:38.287+0000 7f53050fc640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 1 2024-05-08T00:50:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_19.export 2024-05-08T00:50:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_19.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_19.export 2024-05-08T00:50:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_20 2024-05-08T00:50:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_20 snap 2024-05-08T00:50:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:50:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:40.359 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:50:40.366+0000 7facebded640 -1 reset not still connected to 0x562ea1345e10 2024-05-08T00:50:40.359 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:50:40.367+0000 7facebded640 -1 reset not still connected to 0x562ea274e820 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-05-08T00:50:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:50:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-05-08T00:50:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-05-08T00:50:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-08T00:50:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:50:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:50:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:50:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-05-08T00:50:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_20 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_20 2024-05-08T00:50:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_20 ']' 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-05-08T00:50:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-08T00:50:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:50:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-05-08T00:50:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:50:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:50:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-05-08T00:50:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-05-08T00:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_20 2024-05-08T00:50:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-05-08T00:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_20 2024-05-08T00:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-05-08T00:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_20.status 2024-05-08T00:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.status 2024-05-08T00:50:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-05-08T00:50:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.status 2024-05-08T00:50:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9487], [object_number=10, tag_tid=1, entry_tid=9486]]], state=connected] 2024-05-08T00:50:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9487], [object_number=10, tag_tid=1, entry_tid=9486]]], state=connected] 2024-05-08T00:50:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.status 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_20 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_20 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_20.status 2024-05-08T00:50:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.status 2024-05-08T00:50:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-05-08T00:50:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.status 2024-05-08T00:50:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9487], [object_number=10, tag_tid=1, entry_tid=9486]]], state=connected] 2024-05-08T00:50:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9487], [object_number=10, tag_tid=1, entry_tid=9486]]], state=connected] 2024-05-08T00:50:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.status 2024-05-08T00:50:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-05-08T00:50:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:50:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-05-08T00:50:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_20': 2024-05-08T00:50:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad94879bff 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad94879bff 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:44:07 2024 2024-05-08T00:50:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:44:07 2024 2024-05-08T00:50:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:45:08 2024 2024-05-08T00:50:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad94879bff 2024-05-08T00:50:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 7f5aca13-f087-4870-a519-9c11d512ce5a 2024-05-08T00:50:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_20 snap 2024-05-08T00:50:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:50:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.export 2024-05-08T00:50:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_20.export 2024-05-08T00:50:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_20.export 2024-05-08T00:50:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.export 2024-05-08T00:50:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_20.export 2024-05-08T00:50:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_20.export 2024-05-08T00:50:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_20.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_20.export 2024-05-08T00:50:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_21 2024-05-08T00:50:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_21 snap 2024-05-08T00:50:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:50:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-05-08T00:50:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-05-08T00:50:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-05-08T00:50:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-05-08T00:50:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_21 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:50:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_21 2024-05-08T00:50:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:50:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_21 ']' 2024-05-08T00:50:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-05-08T00:50:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-05-08T00:50:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-05-08T00:50:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-05-08T00:50:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-05-08T00:50:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_21 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_21 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_21.status 2024-05-08T00:50:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.status 2024-05-08T00:50:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-05-08T00:50:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.status 2024-05-08T00:50:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7335], [object_number=10, tag_tid=1, entry_tid=7334]]], state=connected] 2024-05-08T00:50:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7335], [object_number=10, tag_tid=1, entry_tid=7334]]], state=connected] 2024-05-08T00:50:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.status 2024-05-08T00:50:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_21 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_21 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_21.status 2024-05-08T00:50:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.status 2024-05-08T00:50:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-05-08T00:50:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.status 2024-05-08T00:50:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:50:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:50:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7335], [object_number=10, tag_tid=1, entry_tid=7334]]], state=connected] 2024-05-08T00:50:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7335], [object_number=10, tag_tid=1, entry_tid=7334]]], state=connected] 2024-05-08T00:50:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.status 2024-05-08T00:50:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-05-08T00:50:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_21': 2024-05-08T00:50:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa57d3ddf1 2024-05-08T00:50:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa57d3ddf1 2024-05-08T00:50:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:44:18 2024 2024-05-08T00:50:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:44:18 2024 2024-05-08T00:50:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:45:19 2024 2024-05-08T00:50:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa57d3ddf1 2024-05-08T00:50:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 0fbec6d5-26fe-4f0f-8fb7-edda6410d8e7 2024-05-08T00:50:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_21 snap 2024-05-08T00:50:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:50:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.export 2024-05-08T00:50:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_21.export 2024-05-08T00:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_21.export 2024-05-08T00:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.export 2024-05-08T00:50:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_21.export 2024-05-08T00:50:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_21.export 2024-05-08T00:50:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_21.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_21.export 2024-05-08T00:50:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_22 2024-05-08T00:50:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_22 snap 2024-05-08T00:50:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:50:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-05-08T00:50:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:50:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-05-08T00:50:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:50:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-05-08T00:50:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:50:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-05-08T00:50:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-05-08T00:50:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-05-08T00:50:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:50:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-05-08T00:50:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:50:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-05-08T00:50:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_22 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:50:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_22 2024-05-08T00:50:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_22 ']' 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-05-08T00:50:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-05-08T00:50:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:50:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:50:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:50:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:50:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:50:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:50:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:50:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:50:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-05-08T00:50:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-05-08T00:50:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_22 2024-05-08T00:50:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-05-08T00:50:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_22 2024-05-08T00:50:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-05-08T00:50:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_22.status 2024-05-08T00:50:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.status 2024-05-08T00:50:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-05-08T00:50:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.status 2024-05-08T00:50:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11659], [object_number=14, tag_tid=1, entry_tid=11658]]], state=connected] 2024-05-08T00:50:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11659], [object_number=14, tag_tid=1, entry_tid=11658]]], state=connected] 2024-05-08T00:50:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.status 2024-05-08T00:50:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:53.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-05-08T00:50:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_22 2024-05-08T00:50:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-05-08T00:50:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_22 2024-05-08T00:50:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-05-08T00:50:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_22.status 2024-05-08T00:50:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.status 2024-05-08T00:50:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-05-08T00:50:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.status 2024-05-08T00:50:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:50:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:50:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11659], [object_number=14, tag_tid=1, entry_tid=11658]]], state=connected] 2024-05-08T00:50:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11659], [object_number=14, tag_tid=1, entry_tid=11658]]], state=connected] 2024-05-08T00:50:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.status 2024-05-08T00:50:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:50:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-05-08T00:50:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:50:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-05-08T00:50:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_22': 2024-05-08T00:50:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ade3d94b79 2024-05-08T00:50:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ade3d94b79 2024-05-08T00:50:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:44:28 2024 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:44:28 2024 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:45:29 2024 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ade3d94b79 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: c59507b5-714a-4ebb-9285-ef2e3100ecff 2024-05-08T00:50:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_22 snap 2024-05-08T00:50:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:50:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.export 2024-05-08T00:50:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_22.export 2024-05-08T00:50:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_22.export 2024-05-08T00:50:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.export 2024-05-08T00:50:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_22.export 2024-05-08T00:50:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_22.export 2024-05-08T00:50:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_22.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_22.export 2024-05-08T00:50:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_23 2024-05-08T00:50:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_23 snap 2024-05-08T00:50:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:50:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:50:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-05-08T00:50:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:50:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-05-08T00:50:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:50:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:50:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-05-08T00:50:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:50:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:50:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:50:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-05-08T00:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:50:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:50:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:50:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:50:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:50:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-05-08T00:50:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:50:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:50:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:50:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:50:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:50:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:50:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:50:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-05-08T00:50:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_23 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:50:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:50:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_23 2024-05-08T00:50:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:50:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:50:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_23 ']' 2024-05-08T00:50:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-05-08T00:50:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-05-08T00:50:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:50:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:50:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:50:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:50:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:50:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:50:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:50:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:50:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-05-08T00:50:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:50:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:50:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:50:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:50:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:50:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:50:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:50:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:50:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-05-08T00:50:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-05-08T00:50:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_23 2024-05-08T00:50:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-05-08T00:50:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_23 2024-05-08T00:50:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:50:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-05-08T00:50:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_23.status 2024-05-08T00:50:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.status 2024-05-08T00:50:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-05-08T00:50:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.status 2024-05-08T00:50:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:50:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:50:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6911], [object_number=6, tag_tid=1, entry_tid=6910]]], state=connected] 2024-05-08T00:50:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6911], [object_number=6, tag_tid=1, entry_tid=6910]]], state=connected] 2024-05-08T00:50:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.status 2024-05-08T00:50:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_23 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_23 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_23.status 2024-05-08T00:50:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:50:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.status 2024-05-08T00:50:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-05-08T00:50:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.status 2024-05-08T00:50:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:50:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:50:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:50:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6911], [object_number=6, tag_tid=1, entry_tid=6910]]], state=connected] 2024-05-08T00:50:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6911], [object_number=6, tag_tid=1, entry_tid=6910]]], state=connected] 2024-05-08T00:50:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.status 2024-05-08T00:50:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:50:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:50:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-05-08T00:50:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:50:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:50:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:50:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:50:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:50:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_23': 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aad143eebd 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aad143eebd 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:50:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:44:38 2024 2024-05-08T00:50:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:44:38 2024 2024-05-08T00:50:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:45:39 2024 2024-05-08T00:50:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:50:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aad143eebd 2024-05-08T00:50:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:50:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:50:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 50d1454d-48f1-440f-b63d-104eaf8cd0fc 2024-05-08T00:50:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:50:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_23 snap 2024-05-08T00:50:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:50:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:50:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.export 2024-05-08T00:50:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_23.export 2024-05-08T00:50:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_23.export 2024-05-08T00:50:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.export 2024-05-08T00:50:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_23.export 2024-05-08T00:50:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:50:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_23.export 2024-05-08T00:50:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_23.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_23.export 2024-05-08T00:50:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:50:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:50:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_24 2024-05-08T00:50:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_24 snap 2024-05-08T00:50:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:50:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:50:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:50:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:50:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-05-08T00:51:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:51:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-05-08T00:51:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:51:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-05-08T00:51:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:51:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-05-08T00:51:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-05-08T00:51:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-05-08T00:51:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-05-08T00:51:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:51:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-05-08T00:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-05-08T00:51:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_24 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_24 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_24 ']' 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-05-08T00:51:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:04.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-05-08T00:51:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-05-08T00:51:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:51:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-05-08T00:51:05.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_24 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_24 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_24.status 2024-05-08T00:51:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.status 2024-05-08T00:51:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-05-08T00:51:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.status 2024-05-08T00:51:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10539], [object_number=10, tag_tid=1, entry_tid=10538]]], state=connected] 2024-05-08T00:51:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10539], [object_number=10, tag_tid=1, entry_tid=10538]]], state=connected] 2024-05-08T00:51:05.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.status 2024-05-08T00:51:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_24 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_24 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_24.status 2024-05-08T00:51:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.status 2024-05-08T00:51:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-05-08T00:51:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.status 2024-05-08T00:51:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10539], [object_number=10, tag_tid=1, entry_tid=10538]]], state=connected] 2024-05-08T00:51:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10539], [object_number=10, tag_tid=1, entry_tid=10538]]], state=connected] 2024-05-08T00:51:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.status 2024-05-08T00:51:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-05-08T00:51:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:51:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-05-08T00:51:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_24': 2024-05-08T00:51:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10adc378475 2024-05-08T00:51:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10adc378475 2024-05-08T00:51:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:44:46 2024 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:44:46 2024 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:45:47 2024 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10adc378475 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 4081a9a3-722f-4583-aba1-9c9817808e58 2024-05-08T00:51:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_24 snap 2024-05-08T00:51:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:51:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.export 2024-05-08T00:51:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_24.export 2024-05-08T00:51:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_24.export 2024-05-08T00:51:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.export 2024-05-08T00:51:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_24.export 2024-05-08T00:51:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_24.export 2024-05-08T00:51:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_24.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_24.export 2024-05-08T00:51:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_25 2024-05-08T00:51:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_25 snap 2024-05-08T00:51:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:51:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:08.808 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:51:08.816+0000 7fa2f29c3640 -1 reset not still connected to 0x5640d3f12c30 2024-05-08T00:51:08.808 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:51:08.816+0000 7fa2f29c3640 -1 reset not still connected to 0x5640e0681ee0 2024-05-08T00:51:08.808 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:51:08.816+0000 7fa2f29c3640 -1 reset not still connected to 0x5640e19a6750 2024-05-08T00:51:08.808 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:51:08.816+0000 7fa2f29c3640 -1 reset not still connected to 0x5640ea004750 2024-05-08T00:51:08.808 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:51:08.816+0000 7fa2f29c3640 -1 reset not still connected to 0x5640ed0a7110 2024-05-08T00:51:08.809 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:51:08.816+0000 7fa2f29c3640 -1 reset not still connected to 0x5640edd8e750 2024-05-08T00:51:08.809 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:51:08.816+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f23055f0 2024-05-08T00:51:08.809 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:51:08.816+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f2eabd40 2024-05-08T00:51:08.809 INFO:tasks.ceph.cluster2.osd.0.smithi116.stderr:2024-05-08T00:51:08.816+0000 7fa2f29c3640 -1 reset not still connected to 0x5640f34e0b60 2024-05-08T00:51:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-05-08T00:51:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:51:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-05-08T00:51:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:51:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-05-08T00:51:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:51:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:10.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:10.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:10.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:10.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-05-08T00:51:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-05-08T00:51:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:51:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_25 2024-05-08T00:51:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:51:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_25 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_25 ']' 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-05-08T00:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-05-08T00:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_25 2024-05-08T00:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-05-08T00:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_25 2024-05-08T00:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-05-08T00:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_25.status 2024-05-08T00:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.status 2024-05-08T00:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-05-08T00:51:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.status 2024-05-08T00:51:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9094], [object_number=11, tag_tid=1, entry_tid=9091]]], state=connected] 2024-05-08T00:51:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9094], [object_number=11, tag_tid=1, entry_tid=9091]]], state=connected] 2024-05-08T00:51:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.status 2024-05-08T00:51:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_25 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_25 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-05-08T00:51:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_25.status 2024-05-08T00:51:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.status 2024-05-08T00:51:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-05-08T00:51:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.status 2024-05-08T00:51:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9094], [object_number=11, tag_tid=1, entry_tid=9091]]], state=connected] 2024-05-08T00:51:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9094], [object_number=11, tag_tid=1, entry_tid=9091]]], state=connected] 2024-05-08T00:51:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.status 2024-05-08T00:51:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-05-08T00:51:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:51:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-05-08T00:51:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_25': 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa739f3ebb 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa739f3ebb 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:44:56 2024 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:44:56 2024 2024-05-08T00:51:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:45:57 2024 2024-05-08T00:51:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa739f3ebb 2024-05-08T00:51:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 06220bb0-67ca-4054-af77-2069341b4f41 2024-05-08T00:51:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_25 snap 2024-05-08T00:51:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:51:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.export 2024-05-08T00:51:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_25.export 2024-05-08T00:51:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_25.export 2024-05-08T00:51:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.export 2024-05-08T00:51:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_25.export 2024-05-08T00:51:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_25.export 2024-05-08T00:51:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_25.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_25.export 2024-05-08T00:51:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_26 2024-05-08T00:51:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_26 snap 2024-05-08T00:51:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:51:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-05-08T00:51:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-05-08T00:51:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-05-08T00:51:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:51:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:51:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-05-08T00:51:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-05-08T00:51:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_26 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_26 2024-05-08T00:51:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:51:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_26 ']' 2024-05-08T00:51:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_26' 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2024-05-08T00:51:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-05-08T00:51:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:51:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:51:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2024-05-08T00:51:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_26 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_26 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_26 '' 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_26 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_26.status 2024-05-08T00:51:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.status 2024-05-08T00:51:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-05-08T00:51:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.status 2024-05-08T00:51:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:51:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:51:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.status 2024-05-08T00:51:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2024-05-08T00:51:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_26 2024-05-08T00:51:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2024-05-08T00:51:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_26 2024-05-08T00:51:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-05-08T00:51:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_26.status 2024-05-08T00:51:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.status 2024-05-08T00:51:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-05-08T00:51:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.status 2024-05-08T00:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:51:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:51:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.status 2024-05-08T00:51:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-05-08T00:51:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:51:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_26': 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10adb91f5e84 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10adb91f5e84 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:45:05 2024 2024-05-08T00:51:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:45:05 2024 2024-05-08T00:51:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:46:06 2024 2024-05-08T00:51:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10adb91f5e84 2024-05-08T00:51:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 1e610a8c-f866-4459-9030-f6a144d3605a 2024-05-08T00:51:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_26 snap 2024-05-08T00:51:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:51:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.export 2024-05-08T00:51:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_26.export 2024-05-08T00:51:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_26.export 2024-05-08T00:51:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.export 2024-05-08T00:51:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_26.export 2024-05-08T00:51:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_26.export 2024-05-08T00:51:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_26.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_26.export 2024-05-08T00:51:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_27 2024-05-08T00:51:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_27 snap 2024-05-08T00:51:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:51:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-05-08T00:51:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:51:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-05-08T00:51:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:51:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-05-08T00:51:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:51:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-05-08T00:51:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-05-08T00:51:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_27 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_27 2024-05-08T00:51:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_27 ']' 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-05-08T00:51:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-05-08T00:51:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_27 2024-05-08T00:51:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-05-08T00:51:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_27 2024-05-08T00:51:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-05-08T00:51:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_27.status 2024-05-08T00:51:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.status 2024-05-08T00:51:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-05-08T00:51:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.status 2024-05-08T00:51:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7987], [object_number=10, tag_tid=1, entry_tid=7986]]], state=connected] 2024-05-08T00:51:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7987], [object_number=10, tag_tid=1, entry_tid=7986]]], state=connected] 2024-05-08T00:51:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.status 2024-05-08T00:51:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-05-08T00:51:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_27 2024-05-08T00:51:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-05-08T00:51:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_27 2024-05-08T00:51:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-05-08T00:51:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_27.status 2024-05-08T00:51:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.status 2024-05-08T00:51:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-05-08T00:51:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.status 2024-05-08T00:51:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7987], [object_number=10, tag_tid=1, entry_tid=7986]]], state=connected] 2024-05-08T00:51:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=7987], [object_number=10, tag_tid=1, entry_tid=7986]]], state=connected] 2024-05-08T00:51:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.status 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-05-08T00:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_27': 2024-05-08T00:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa1afdd9b8 2024-05-08T00:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa1afdd9b8 2024-05-08T00:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:45:10 2024 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:45:10 2024 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:46:11 2024 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa1afdd9b8 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: aa9392de-0874-4e36-8d6a-4902138153dd 2024-05-08T00:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_27 snap 2024-05-08T00:51:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:51:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.export 2024-05-08T00:51:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_27.export 2024-05-08T00:51:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_27.export 2024-05-08T00:51:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.export 2024-05-08T00:51:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_27.export 2024-05-08T00:51:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_27.export 2024-05-08T00:51:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_27.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_27.export 2024-05-08T00:51:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_28 2024-05-08T00:51:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_28 snap 2024-05-08T00:51:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:51:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-05-08T00:51:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:51:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-05-08T00:51:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:51:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-05-08T00:51:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:51:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-05-08T00:51:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-05-08T00:51:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-05-08T00:51:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-05-08T00:51:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:51:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-05-08T00:51:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:51:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:51:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:51:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-05-08T00:51:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-05-08T00:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_28 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_28 2024-05-08T00:51:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:51:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_28 ']' 2024-05-08T00:51:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-05-08T00:51:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-05-08T00:51:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-05-08T00:51:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-05-08T00:51:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-05-08T00:51:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:51:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-05-08T00:51:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:51:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-05-08T00:51:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-05-08T00:51:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_28 2024-05-08T00:51:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-05-08T00:51:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_28 2024-05-08T00:51:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-05-08T00:51:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_28.status 2024-05-08T00:51:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.status 2024-05-08T00:51:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-05-08T00:51:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.status 2024-05-08T00:51:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8055], [object_number=10, tag_tid=1, entry_tid=8054]]], state=connected] 2024-05-08T00:51:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8055], [object_number=10, tag_tid=1, entry_tid=8054]]], state=connected] 2024-05-08T00:51:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.status 2024-05-08T00:51:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_28 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_28 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-05-08T00:51:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_28.status 2024-05-08T00:51:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.status 2024-05-08T00:51:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-05-08T00:51:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.status 2024-05-08T00:51:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8055], [object_number=10, tag_tid=1, entry_tid=8054]]], state=connected] 2024-05-08T00:51:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8055], [object_number=10, tag_tid=1, entry_tid=8054]]], state=connected] 2024-05-08T00:51:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.status 2024-05-08T00:51:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-05-08T00:51:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:51:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-05-08T00:51:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_28': 2024-05-08T00:51:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad15d54393 2024-05-08T00:51:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad15d54393 2024-05-08T00:51:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:45:19 2024 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:45:19 2024 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:46:20 2024 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad15d54393 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 824a1dfe-5a28-477d-8a35-5a650202eae2 2024-05-08T00:51:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_28 snap 2024-05-08T00:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.export 2024-05-08T00:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_28.export 2024-05-08T00:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_28.export 2024-05-08T00:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.export 2024-05-08T00:51:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_28.export 2024-05-08T00:51:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_28.export 2024-05-08T00:51:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_28.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_28.export 2024-05-08T00:51:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_29 2024-05-08T00:51:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_29 snap 2024-05-08T00:51:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:51:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-05-08T00:51:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-05-08T00:51:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-05-08T00:51:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-05-08T00:51:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_29 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_29 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_29 ']' 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-05-08T00:51:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-05-08T00:51:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-05-08T00:51:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_29 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_29 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_29.status 2024-05-08T00:51:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.status 2024-05-08T00:51:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-05-08T00:51:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.status 2024-05-08T00:51:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5155], [object_number=6, tag_tid=1, entry_tid=5154]]], state=connected] 2024-05-08T00:51:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5155], [object_number=6, tag_tid=1, entry_tid=5154]]], state=connected] 2024-05-08T00:51:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.status 2024-05-08T00:51:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-05-08T00:51:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_29 2024-05-08T00:51:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-05-08T00:51:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_29 2024-05-08T00:51:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-05-08T00:51:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_29.status 2024-05-08T00:51:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.status 2024-05-08T00:51:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-05-08T00:51:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.status 2024-05-08T00:51:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5155], [object_number=6, tag_tid=1, entry_tid=5154]]], state=connected] 2024-05-08T00:51:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5155], [object_number=6, tag_tid=1, entry_tid=5154]]], state=connected] 2024-05-08T00:51:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.status 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-05-08T00:51:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_29': 2024-05-08T00:51:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa6ab9ab7d 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa6ab9ab7d 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:45:27 2024 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:45:27 2024 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:46:28 2024 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa6ab9ab7d 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 80239aed-452b-47d2-9665-5c4fbb826f66 2024-05-08T00:51:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_29 snap 2024-05-08T00:51:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:51:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.export 2024-05-08T00:51:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_29.export 2024-05-08T00:51:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_29.export 2024-05-08T00:51:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.export 2024-05-08T00:51:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_29.export 2024-05-08T00:51:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_29.export 2024-05-08T00:51:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_29.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_29.export 2024-05-08T00:51:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_30 2024-05-08T00:51:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_30 snap 2024-05-08T00:51:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:51:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-05-08T00:51:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:51:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-05-08T00:51:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:51:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-05-08T00:51:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:51:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-05-08T00:51:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-05-08T00:51:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-05-08T00:51:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:51:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-05-08T00:51:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:51:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:51:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:51:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-05-08T00:51:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_30 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_30 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_30 ']' 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-05-08T00:51:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-05-08T00:51:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-05-08T00:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:51:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-05-08T00:51:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:51:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-05-08T00:51:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-05-08T00:51:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_30 2024-05-08T00:51:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-05-08T00:51:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_30 2024-05-08T00:51:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-05-08T00:51:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_30.status 2024-05-08T00:51:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.status 2024-05-08T00:51:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-05-08T00:51:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.status 2024-05-08T00:51:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7054], [object_number=11, tag_tid=1, entry_tid=7051]]], state=connected] 2024-05-08T00:51:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7054], [object_number=11, tag_tid=1, entry_tid=7051]]], state=connected] 2024-05-08T00:51:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.status 2024-05-08T00:51:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_30 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_30 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_30.status 2024-05-08T00:51:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.status 2024-05-08T00:51:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-05-08T00:51:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.status 2024-05-08T00:51:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7054], [object_number=11, tag_tid=1, entry_tid=7051]]], state=connected] 2024-05-08T00:51:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7054], [object_number=11, tag_tid=1, entry_tid=7051]]], state=connected] 2024-05-08T00:51:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.status 2024-05-08T00:51:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:36.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_30': 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10adf39b31b8 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10adf39b31b8 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:45:33 2024 2024-05-08T00:51:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:45:33 2024 2024-05-08T00:51:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:46:34 2024 2024-05-08T00:51:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10adf39b31b8 2024-05-08T00:51:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 70155194-7fc8-417c-9f93-810a952a4a46 2024-05-08T00:51:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_30 snap 2024-05-08T00:51:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:51:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.export 2024-05-08T00:51:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_30.export 2024-05-08T00:51:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_30.export 2024-05-08T00:51:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.export 2024-05-08T00:51:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_30.export 2024-05-08T00:51:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_30.export 2024-05-08T00:51:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_30.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_30.export 2024-05-08T00:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_31 2024-05-08T00:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_31 snap 2024-05-08T00:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:37.518 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:51:37.528+0000 7f9ff5d0e640 -1 bdev(0x5564eb38ce00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2024-05-08T00:51:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-05-08T00:51:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:51:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-05-08T00:51:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:51:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-05-08T00:51:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-05-08T00:51:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:39.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-05-08T00:51:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_31 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_31 2024-05-08T00:51:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_31 ']' 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-05-08T00:51:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-05-08T00:51:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_31 2024-05-08T00:51:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-05-08T00:51:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_31 2024-05-08T00:51:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-05-08T00:51:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_31.status 2024-05-08T00:51:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.status 2024-05-08T00:51:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-05-08T00:51:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.status 2024-05-08T00:51:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6651], [object_number=6, tag_tid=1, entry_tid=6650]]], state=connected] 2024-05-08T00:51:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6651], [object_number=6, tag_tid=1, entry_tid=6650]]], state=connected] 2024-05-08T00:51:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.status 2024-05-08T00:51:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:51:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-05-08T00:51:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_31 2024-05-08T00:51:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-05-08T00:51:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_31 2024-05-08T00:51:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-05-08T00:51:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_31.status 2024-05-08T00:51:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.status 2024-05-08T00:51:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-05-08T00:51:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.status 2024-05-08T00:51:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6651], [object_number=6, tag_tid=1, entry_tid=6650]]], state=connected] 2024-05-08T00:51:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6651], [object_number=6, tag_tid=1, entry_tid=6650]]], state=connected] 2024-05-08T00:51:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.status 2024-05-08T00:51:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:51:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:51:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-05-08T00:51:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:51:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-05-08T00:51:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_31': 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aabd1b4210 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aabd1b4210 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:45:41 2024 2024-05-08T00:51:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:45:41 2024 2024-05-08T00:51:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:46:42 2024 2024-05-08T00:51:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aabd1b4210 2024-05-08T00:51:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 54baf757-455d-4f6d-8eab-9a3b1947188c 2024-05-08T00:51:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_31 snap 2024-05-08T00:51:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:51:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.export 2024-05-08T00:51:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_31.export 2024-05-08T00:51:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_31.export 2024-05-08T00:51:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.export 2024-05-08T00:51:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_31.export 2024-05-08T00:51:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_31.export 2024-05-08T00:51:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_31.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_31.export 2024-05-08T00:51:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_32 2024-05-08T00:51:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_32 snap 2024-05-08T00:51:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:51:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-05-08T00:51:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:51:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-05-08T00:51:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-05-08T00:51:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-05-08T00:51:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:51:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:51:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-05-08T00:51:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:43.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_32 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:51:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_32 2024-05-08T00:51:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:51:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_32 ']' 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-05-08T00:51:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-05-08T00:51:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-05-08T00:51:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:51:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-05-08T00:51:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:44.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_32 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_32 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-05-08T00:51:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_32.status 2024-05-08T00:51:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.status 2024-05-08T00:51:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-05-08T00:51:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.status 2024-05-08T00:51:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4602], [object_number=7, tag_tid=1, entry_tid=4599]]], state=connected] 2024-05-08T00:51:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4602], [object_number=7, tag_tid=1, entry_tid=4599]]], state=connected] 2024-05-08T00:51:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.status 2024-05-08T00:51:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-05-08T00:51:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_32 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_32 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_32.status 2024-05-08T00:51:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.status 2024-05-08T00:51:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-05-08T00:51:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.status 2024-05-08T00:51:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4602], [object_number=7, tag_tid=1, entry_tid=4599]]], state=connected] 2024-05-08T00:51:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=4602], [object_number=7, tag_tid=1, entry_tid=4599]]], state=connected] 2024-05-08T00:51:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.status 2024-05-08T00:51:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-05-08T00:51:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:51:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-05-08T00:51:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_32': 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10add9198b9f 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10add9198b9f 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:45:48 2024 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:45:48 2024 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:46:49 2024 2024-05-08T00:51:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10add9198b9f 2024-05-08T00:51:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 3aaa9955-d806-4b32-bc85-772484ad63fc 2024-05-08T00:51:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_32 snap 2024-05-08T00:51:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:51:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.export 2024-05-08T00:51:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_32.export 2024-05-08T00:51:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_32.export 2024-05-08T00:51:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.export 2024-05-08T00:51:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_32.export 2024-05-08T00:51:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_32.export 2024-05-08T00:51:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_32.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_32.export 2024-05-08T00:51:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_33 2024-05-08T00:51:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_33 snap 2024-05-08T00:51:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:51:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-05-08T00:51:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-05-08T00:51:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-05-08T00:51:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-05-08T00:51:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_33 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_33 2024-05-08T00:51:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:51:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_33 ']' 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_33 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_33 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-05-08T00:51:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_33.status 2024-05-08T00:51:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.status 2024-05-08T00:51:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-05-08T00:51:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.status 2024-05-08T00:51:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:51:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10282], [object_number=11, tag_tid=1, entry_tid=10279]]], state=connected] 2024-05-08T00:51:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10282], [object_number=11, tag_tid=1, entry_tid=10279]]], state=connected] 2024-05-08T00:51:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.status 2024-05-08T00:51:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:51:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-05-08T00:51:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_33 2024-05-08T00:51:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-05-08T00:51:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_33 2024-05-08T00:51:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-05-08T00:51:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_33.status 2024-05-08T00:51:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.status 2024-05-08T00:51:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-05-08T00:51:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.status 2024-05-08T00:51:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:51:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10282], [object_number=11, tag_tid=1, entry_tid=10279]]], state=connected] 2024-05-08T00:51:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10282], [object_number=11, tag_tid=1, entry_tid=10279]]], state=connected] 2024-05-08T00:51:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.status 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-05-08T00:51:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_33': 2024-05-08T00:51:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa7fb417a8 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa7fb417a8 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:45:53 2024 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:45:53 2024 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:46:54 2024 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa7fb417a8 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 63a6c963-b8df-447b-95ef-257e5597a891 2024-05-08T00:51:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_33 snap 2024-05-08T00:51:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:51:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.export 2024-05-08T00:51:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_33.export 2024-05-08T00:51:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_33.export 2024-05-08T00:51:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.export 2024-05-08T00:51:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_33.export 2024-05-08T00:51:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_33.export 2024-05-08T00:51:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_33.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_33.export 2024-05-08T00:51:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_34 2024-05-08T00:51:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_34 snap 2024-05-08T00:51:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:51:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:51.345 INFO:tasks.ceph.cluster2.osd.1.smithi116.stderr:2024-05-08T00:51:51.355+0000 7facebded640 -1 reset not still connected to 0x562e84f561a0 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-05-08T00:51:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:51:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-05-08T00:51:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-05-08T00:51:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:51:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-05-08T00:51:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:51:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:51:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-05-08T00:51:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-05-08T00:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_34 2024-05-08T00:51:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:51:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_34 2024-05-08T00:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_34 ']' 2024-05-08T00:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-05-08T00:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-05-08T00:51:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-05-08T00:51:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-05-08T00:51:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-05-08T00:51:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:51:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:51:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:51:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:51:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:51:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:51:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:51:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-05-08T00:51:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_34 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_34 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-05-08T00:51:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_34.status 2024-05-08T00:51:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.status 2024-05-08T00:51:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-05-08T00:51:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.status 2024-05-08T00:51:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8547], [object_number=10, tag_tid=1, entry_tid=8546]]], state=connected] 2024-05-08T00:51:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8547], [object_number=10, tag_tid=1, entry_tid=8546]]], state=connected] 2024-05-08T00:51:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.status 2024-05-08T00:51:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_34 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_34 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_34.status 2024-05-08T00:51:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.status 2024-05-08T00:51:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-05-08T00:51:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.status 2024-05-08T00:51:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:51:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:51:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8547], [object_number=10, tag_tid=1, entry_tid=8546]]], state=connected] 2024-05-08T00:51:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8547], [object_number=10, tag_tid=1, entry_tid=8546]]], state=connected] 2024-05-08T00:51:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.status 2024-05-08T00:51:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:51:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-05-08T00:51:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:51:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-05-08T00:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_34': 2024-05-08T00:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad6679445b 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad6679445b 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:46:06 2024 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:46:06 2024 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:47:07 2024 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad6679445b 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: fd006a50-dd7e-4bfc-9437-6ae36384be5a 2024-05-08T00:51:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_34 snap 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.export 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_34.export 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_34.export 2024-05-08T00:51:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.export 2024-05-08T00:51:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_34.export 2024-05-08T00:51:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_34.export 2024-05-08T00:51:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_34.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_34.export 2024-05-08T00:51:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_35 2024-05-08T00:51:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_35 snap 2024-05-08T00:51:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-05-08T00:51:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:51:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:51:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:51:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-05-08T00:51:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:51:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:51:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:51:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:51:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:51:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-05-08T00:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:51:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:51:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:51:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-05-08T00:51:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:51:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:51:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:51:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:51:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:51:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:51:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:51:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-05-08T00:51:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_35 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_35 2024-05-08T00:51:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_35 ']' 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-05-08T00:51:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-05-08T00:51:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_35.status 2024-05-08T00:51:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-05-08T00:51:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3827], [object_number=6, tag_tid=1, entry_tid=3826]]], state=connected] 2024-05-08T00:51:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3827], [object_number=6, tag_tid=1, entry_tid=3826]]], state=connected] 2024-05-08T00:51:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-05-08T00:51:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:51:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-05-08T00:51:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:51:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-05-08T00:51:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_35.status 2024-05-08T00:51:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-05-08T00:51:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3827], [object_number=6, tag_tid=1, entry_tid=3826]]], state=connected] 2024-05-08T00:51:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3827], [object_number=6, tag_tid=1, entry_tid=3826]]], state=connected] 2024-05-08T00:51:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:51:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:51:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-05-08T00:51:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:51:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-05-08T00:51:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:51:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:51:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:51:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=3 2024-05-08T00:51:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:51:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:51:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:51:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=2 2024-05-08T00:51:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:51:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:51:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:51:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:51:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:51:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:51:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=1 2024-05-08T00:51:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 3 = 2 -a 1 -le 1 2024-05-08T00:51:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:51:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:51:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_35 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_35 ']' 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:51:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:51:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:51:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:51:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:51:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:51:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:51:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:51:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-05-08T00:51:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:51:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:51:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:51:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:51:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:51:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:51:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:51:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:51:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-05-08T00:51:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_35.status 2024-05-08T00:51:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-05-08T00:51:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3827], [object_number=6, tag_tid=1, entry_tid=3826]]], state=connected] 2024-05-08T00:51:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3827], [object_number=6, tag_tid=1, entry_tid=3826]]], state=connected] 2024-05-08T00:51:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-05-08T00:51:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_35.status 2024-05-08T00:51:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:51:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-05-08T00:51:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:51:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:51:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:51:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3827], [object_number=6, tag_tid=1, entry_tid=3826]]], state=connected] 2024-05-08T00:51:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=3827], [object_number=6, tag_tid=1, entry_tid=3826]]], state=connected] 2024-05-08T00:51:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.status 2024-05-08T00:51:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:51:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-05-08T00:51:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:51:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:51:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:51:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:51:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_35': 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa7ca105c9 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa7ca105c9 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:51:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:51:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:46:15 2024 2024-05-08T00:51:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:46:15 2024 2024-05-08T00:51:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:47:16 2024 2024-05-08T00:51:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:51:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa7ca105c9 2024-05-08T00:51:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:51:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:51:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: b8f52b64-c082-48e6-ba98-b5d2c7422b79 2024-05-08T00:51:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:51:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_35 snap 2024-05-08T00:51:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:51:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:51:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.export 2024-05-08T00:51:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_35.export 2024-05-08T00:51:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_35.export 2024-05-08T00:51:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.export 2024-05-08T00:51:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_35.export 2024-05-08T00:51:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:51:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_35.export 2024-05-08T00:51:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_35.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_35.export 2024-05-08T00:51:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:51:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:51:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_36 2024-05-08T00:51:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_36 snap 2024-05-08T00:51:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:51:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:51:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:51:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:51:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-05-08T00:52:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:52:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-05-08T00:52:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-05-08T00:52:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-05-08T00:52:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-05-08T00:52:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:52:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:52:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:52:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-05-08T00:52:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_36 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:52:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_36 2024-05-08T00:52:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:52:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_36 ']' 2024-05-08T00:52:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-05-08T00:52:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-05-08T00:52:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-05-08T00:52:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-05-08T00:52:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-05-08T00:52:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:52:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-05-08T00:52:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_36 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_36 2024-05-08T00:52:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-05-08T00:52:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_36.status 2024-05-08T00:52:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.status 2024-05-08T00:52:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-05-08T00:52:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.status 2024-05-08T00:52:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10051], [object_number=10, tag_tid=1, entry_tid=10050]]], state=connected] 2024-05-08T00:52:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10051], [object_number=10, tag_tid=1, entry_tid=10050]]], state=connected] 2024-05-08T00:52:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.status 2024-05-08T00:52:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_36 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_36 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_36.status 2024-05-08T00:52:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.status 2024-05-08T00:52:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-05-08T00:52:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.status 2024-05-08T00:52:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10051], [object_number=10, tag_tid=1, entry_tid=10050]]], state=connected] 2024-05-08T00:52:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=10051], [object_number=10, tag_tid=1, entry_tid=10050]]], state=connected] 2024-05-08T00:52:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.status 2024-05-08T00:52:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-05-08T00:52:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:52:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_36': 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad3997b5c4 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad3997b5c4 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:46:21 2024 2024-05-08T00:52:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:46:21 2024 2024-05-08T00:52:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:47:22 2024 2024-05-08T00:52:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad3997b5c4 2024-05-08T00:52:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 27915b32-ae92-44a7-9dc7-c95eafa1247c 2024-05-08T00:52:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_36 snap 2024-05-08T00:52:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:52:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.export 2024-05-08T00:52:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_36.export 2024-05-08T00:52:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_36.export 2024-05-08T00:52:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.export 2024-05-08T00:52:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_36.export 2024-05-08T00:52:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_36.export 2024-05-08T00:52:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_36.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_36.export 2024-05-08T00:52:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_37 2024-05-08T00:52:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_37 snap 2024-05-08T00:52:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-05-08T00:52:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-05-08T00:52:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-05-08T00:52:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-05-08T00:52:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-05-08T00:52:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-05-08T00:52:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_37 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_37 2024-05-08T00:52:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_37 ']' 2024-05-08T00:52:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-05-08T00:52:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-05-08T00:52:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-05-08T00:52:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-05-08T00:52:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-05-08T00:52:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:52:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-05-08T00:52:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:52:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-05-08T00:52:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_37.status 2024-05-08T00:52:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-05-08T00:52:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 0 2024-05-08T00:52:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3375], [object_number=2, tag_tid=1, entry_tid=3374]]], state=connected] 2024-05-08T00:52:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3375], [object_number=2, tag_tid=1, entry_tid=3374]]], state=connected] 2024-05-08T00:52:08.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_37.status 2024-05-08T00:52:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-05-08T00:52:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 0 2024-05-08T00:52:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3375], [object_number=2, tag_tid=1, entry_tid=3374]]], state=connected] 2024-05-08T00:52:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3375], [object_number=2, tag_tid=1, entry_tid=3374]]], state=connected] 2024-05-08T00:52:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-05-08T00:52:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:52:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-05-08T00:52:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:52:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=3 2024-05-08T00:52:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:52:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=2 2024-05-08T00:52:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:52:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:52:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:52:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=1 2024-05-08T00:52:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 3 = 2 -a 1 -le 1 2024-05-08T00:52:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_37 2024-05-08T00:52:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_37 ']' 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-05-08T00:52:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-05-08T00:52:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:52:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-05-08T00:52:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:52:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-05-08T00:52:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_37.status 2024-05-08T00:52:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-05-08T00:52:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 0 2024-05-08T00:52:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3375], [object_number=2, tag_tid=1, entry_tid=3374]]], state=connected] 2024-05-08T00:52:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3375], [object_number=2, tag_tid=1, entry_tid=3374]]], state=connected] 2024-05-08T00:52:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-05-08T00:52:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:52:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-05-08T00:52:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:52:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-05-08T00:52:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_37.status 2024-05-08T00:52:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-05-08T00:52:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 0 2024-05-08T00:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3375], [object_number=2, tag_tid=1, entry_tid=3374]]], state=connected] 2024-05-08T00:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3375], [object_number=2, tag_tid=1, entry_tid=3374]]], state=connected] 2024-05-08T00:52:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.status 2024-05-08T00:52:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-05-08T00:52:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-05-08T00:52:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_37': 2024-05-08T00:52:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa4272b0ec 2024-05-08T00:52:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa4272b0ec 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:46:32 2024 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:46:32 2024 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:47:33 2024 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa4272b0ec 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 2d84c06c-32f4-44b3-9179-6add7ee1a378 2024-05-08T00:52:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_37 snap 2024-05-08T00:52:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:52:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.export 2024-05-08T00:52:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_37.export 2024-05-08T00:52:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_37.export 2024-05-08T00:52:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.export 2024-05-08T00:52:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_37.export 2024-05-08T00:52:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_37.export 2024-05-08T00:52:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_37.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_37.export 2024-05-08T00:52:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_38 2024-05-08T00:52:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_38 snap 2024-05-08T00:52:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:52:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-05-08T00:52:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:52:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-05-08T00:52:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:52:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-05-08T00:52:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-05-08T00:52:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-05-08T00:52:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-05-08T00:52:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:52:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-05-08T00:52:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:52:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:52:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:52:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:52:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-05-08T00:52:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_38 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_38 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_38 ']' 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-05-08T00:52:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-05-08T00:52:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-05-08T00:52:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:52:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-05-08T00:52:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:52:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:52:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-05-08T00:52:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_38 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_38 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_38.status 2024-05-08T00:52:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.status 2024-05-08T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-05-08T00:52:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.status 2024-05-08T00:52:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6910], [object_number=7, tag_tid=1, entry_tid=6907]]], state=connected] 2024-05-08T00:52:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6910], [object_number=7, tag_tid=1, entry_tid=6907]]], state=connected] 2024-05-08T00:52:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.status 2024-05-08T00:52:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-05-08T00:52:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_38 2024-05-08T00:52:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-05-08T00:52:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_38 2024-05-08T00:52:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-05-08T00:52:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_38.status 2024-05-08T00:52:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.status 2024-05-08T00:52:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-05-08T00:52:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.status 2024-05-08T00:52:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6910], [object_number=7, tag_tid=1, entry_tid=6907]]], state=connected] 2024-05-08T00:52:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=6910], [object_number=7, tag_tid=1, entry_tid=6907]]], state=connected] 2024-05-08T00:52:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.status 2024-05-08T00:52:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-05-08T00:52:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:52:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-05-08T00:52:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_38': 2024-05-08T00:52:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad7026ab3b 2024-05-08T00:52:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad7026ab3b 2024-05-08T00:52:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:46:37 2024 2024-05-08T00:52:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:46:37 2024 2024-05-08T00:52:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:47:38 2024 2024-05-08T00:52:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad7026ab3b 2024-05-08T00:52:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 727c3f62-bf7f-432b-af00-823abceeb601 2024-05-08T00:52:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_38 snap 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.export 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_38.export 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_38.export 2024-05-08T00:52:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.export 2024-05-08T00:52:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_38.export 2024-05-08T00:52:14.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_38.export 2024-05-08T00:52:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_38.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_38.export 2024-05-08T00:52:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_39 2024-05-08T00:52:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_39 snap 2024-05-08T00:52:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-05-08T00:52:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-05-08T00:52:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-05-08T00:52:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-05-08T00:52:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-05-08T00:52:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-05-08T00:52:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_39 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_39 2024-05-08T00:52:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_39 ']' 2024-05-08T00:52:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-05-08T00:52:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-05-08T00:52:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-05-08T00:52:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-05-08T00:52:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_39.status 2024-05-08T00:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-05-08T00:52:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4779], [object_number=6, tag_tid=1, entry_tid=4778]]], state=connected] 2024-05-08T00:52:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4779], [object_number=6, tag_tid=1, entry_tid=4778]]], state=connected] 2024-05-08T00:52:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-05-08T00:52:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-05-08T00:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-05-08T00:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_39.status 2024-05-08T00:52:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-05-08T00:52:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4779], [object_number=6, tag_tid=1, entry_tid=4778]]], state=connected] 2024-05-08T00:52:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4779], [object_number=6, tag_tid=1, entry_tid=4778]]], state=connected] 2024-05-08T00:52:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-05-08T00:52:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:52:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-05-08T00:52:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:52:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=3 2024-05-08T00:52:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:52:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=2 2024-05-08T00:52:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:52:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:52:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:52:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=1 2024-05-08T00:52:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 3 = 2 -a 1 -le 1 2024-05-08T00:52:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_39 2024-05-08T00:52:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_39 ']' 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-05-08T00:52:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-05-08T00:52:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-05-08T00:52:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_39.status 2024-05-08T00:52:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-05-08T00:52:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4779], [object_number=6, tag_tid=1, entry_tid=4778]]], state=connected] 2024-05-08T00:52:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4779], [object_number=6, tag_tid=1, entry_tid=4778]]], state=connected] 2024-05-08T00:52:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-05-08T00:52:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_39.status 2024-05-08T00:52:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-05-08T00:52:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4779], [object_number=6, tag_tid=1, entry_tid=4778]]], state=connected] 2024-05-08T00:52:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4779], [object_number=6, tag_tid=1, entry_tid=4778]]], state=connected] 2024-05-08T00:52:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.status 2024-05-08T00:52:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_39': 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa8efb0ffc 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa8efb0ffc 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:46:46 2024 2024-05-08T00:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:46:46 2024 2024-05-08T00:52:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:47:47 2024 2024-05-08T00:52:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa8efb0ffc 2024-05-08T00:52:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 734380a9-939c-4c71-81be-2b79e6863ea8 2024-05-08T00:52:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_39 snap 2024-05-08T00:52:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:52:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.export 2024-05-08T00:52:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_39.export 2024-05-08T00:52:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_39.export 2024-05-08T00:52:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.export 2024-05-08T00:52:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_39.export 2024-05-08T00:52:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_39.export 2024-05-08T00:52:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_39.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_39.export 2024-05-08T00:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_40 2024-05-08T00:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_40 snap 2024-05-08T00:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-05-08T00:52:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-05-08T00:52:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-05-08T00:52:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:52:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:52:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:52:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-05-08T00:52:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_40 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:52:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_40 2024-05-08T00:52:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_40 ']' 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-05-08T00:52:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-05-08T00:52:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:52:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-05-08T00:52:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:52:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-05-08T00:52:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-05-08T00:52:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_40 2024-05-08T00:52:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-05-08T00:52:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_40 2024-05-08T00:52:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-05-08T00:52:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_40.status 2024-05-08T00:52:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.status 2024-05-08T00:52:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-05-08T00:52:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.status 2024-05-08T00:52:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8070], [object_number=11, tag_tid=1, entry_tid=8067]]], state=connected] 2024-05-08T00:52:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8070], [object_number=11, tag_tid=1, entry_tid=8067]]], state=connected] 2024-05-08T00:52:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.status 2024-05-08T00:52:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-05-08T00:52:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_40 2024-05-08T00:52:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-05-08T00:52:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_40 2024-05-08T00:52:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-05-08T00:52:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_40.status 2024-05-08T00:52:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.status 2024-05-08T00:52:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-05-08T00:52:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.status 2024-05-08T00:52:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8070], [object_number=11, tag_tid=1, entry_tid=8067]]], state=connected] 2024-05-08T00:52:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=8070], [object_number=11, tag_tid=1, entry_tid=8067]]], state=connected] 2024-05-08T00:52:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.status 2024-05-08T00:52:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-05-08T00:52:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:52:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-05-08T00:52:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_40': 2024-05-08T00:52:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad92696e16 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad92696e16 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:46:49 2024 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:46:49 2024 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:47:50 2024 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad92696e16 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: f5c4a247-86ec-411f-a257-6db9e0d095cc 2024-05-08T00:52:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_40 snap 2024-05-08T00:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.export 2024-05-08T00:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_40.export 2024-05-08T00:52:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_40.export 2024-05-08T00:52:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.export 2024-05-08T00:52:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_40.export 2024-05-08T00:52:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_40.export 2024-05-08T00:52:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_40.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_40.export 2024-05-08T00:52:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_41 2024-05-08T00:52:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_41 snap 2024-05-08T00:52:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:52:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:25.640 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:52:25.651+0000 7f9ffa517640 -1 bdev(0x5564eb38ce00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 4 2024-05-08T00:52:25.642 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:52:25.652+0000 7f9ff550d640 -1 bdev(0x5564eb38ce00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 7 2024-05-08T00:52:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-05-08T00:52:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:52:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-05-08T00:52:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:52:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-05-08T00:52:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_41 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_41 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_41 ']' 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_41 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_41 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-05-08T00:52:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_41.status 2024-05-08T00:52:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.status 2024-05-08T00:52:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-05-08T00:52:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.status 2024-05-08T00:52:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 0 2024-05-08T00:52:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3427], [object_number=2, tag_tid=1, entry_tid=3426]]], state=connected] 2024-05-08T00:52:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3427], [object_number=2, tag_tid=1, entry_tid=3426]]], state=connected] 2024-05-08T00:52:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.status 2024-05-08T00:52:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_41 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_41 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_41.status 2024-05-08T00:52:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.status 2024-05-08T00:52:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-05-08T00:52:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.status 2024-05-08T00:52:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 0 2024-05-08T00:52:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3427], [object_number=2, tag_tid=1, entry_tid=3426]]], state=connected] 2024-05-08T00:52:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3427], [object_number=2, tag_tid=1, entry_tid=3426]]], state=connected] 2024-05-08T00:52:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.status 2024-05-08T00:52:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=2, entry_tid=1' -a 'object_number=5, tag_tid=2, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-05-08T00:52:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-05-08T00:52:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:52:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-05-08T00:52:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_41': 2024-05-08T00:52:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa2a63aba7 2024-05-08T00:52:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa2a63aba7 2024-05-08T00:52:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:46:59 2024 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:46:59 2024 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:46:59 2024 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa2a63aba7 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 2d0fcb7b-4ffd-4307-af4b-a54a8ae53585 2024-05-08T00:52:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_41 snap 2024-05-08T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.export 2024-05-08T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_41.export 2024-05-08T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_41.export 2024-05-08T00:52:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.export 2024-05-08T00:52:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_41.export 2024-05-08T00:52:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_41.export 2024-05-08T00:52:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_41.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_41.export 2024-05-08T00:52:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_42 2024-05-08T00:52:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_42 snap 2024-05-08T00:52:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:52:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-05-08T00:52:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:52:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-05-08T00:52:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:52:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-05-08T00:52:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-05-08T00:52:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-05-08T00:52:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:52:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:52:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:52:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-05-08T00:52:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_42 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:52:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_42 2024-05-08T00:52:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:52:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_42 ']' 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:32.244 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:32.244 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-05-08T00:52:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-05-08T00:52:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:52:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-05-08T00:52:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:52:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-05-08T00:52:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-05-08T00:52:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_42 2024-05-08T00:52:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-05-08T00:52:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_42 2024-05-08T00:52:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-05-08T00:52:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_42.status 2024-05-08T00:52:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.status 2024-05-08T00:52:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-05-08T00:52:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.status 2024-05-08T00:52:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=11, tag_tid=1, entry_tid=9787]]], state=connected] 2024-05-08T00:52:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=11, tag_tid=1, entry_tid=9787]]], state=connected] 2024-05-08T00:52:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.status 2024-05-08T00:52:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_42 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_42 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_42.status 2024-05-08T00:52:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.status 2024-05-08T00:52:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-05-08T00:52:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.status 2024-05-08T00:52:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=11, tag_tid=1, entry_tid=9787]]], state=connected] 2024-05-08T00:52:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9790], [object_number=11, tag_tid=1, entry_tid=9787]]], state=connected] 2024-05-08T00:52:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.status 2024-05-08T00:52:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-05-08T00:52:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_42': 2024-05-08T00:52:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad177a0e7b 2024-05-08T00:52:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad177a0e7b 2024-05-08T00:52:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:47:04 2024 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:47:04 2024 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:48:05 2024 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad177a0e7b 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 7ff5fdfe-dd34-49d4-b074-674e80a07a12 2024-05-08T00:52:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_42 snap 2024-05-08T00:52:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:52:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.export 2024-05-08T00:52:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_42.export 2024-05-08T00:52:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_42.export 2024-05-08T00:52:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.export 2024-05-08T00:52:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_42.export 2024-05-08T00:52:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_42.export 2024-05-08T00:52:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_42.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_42.export 2024-05-08T00:52:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_43 2024-05-08T00:52:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_43 snap 2024-05-08T00:52:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:52:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-05-08T00:52:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:52:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-05-08T00:52:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:52:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-05-08T00:52:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:52:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-05-08T00:52:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-05-08T00:52:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-05-08T00:52:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_43 2024-05-08T00:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:52:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_43 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_43 ']' 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-05-08T00:52:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-05-08T00:52:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_43 2024-05-08T00:52:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-05-08T00:52:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_43 2024-05-08T00:52:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-05-08T00:52:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_43.status 2024-05-08T00:52:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.status 2024-05-08T00:52:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-05-08T00:52:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.status 2024-05-08T00:52:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:52:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:52:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11747], [object_number=14, tag_tid=1, entry_tid=11746]]], state=connected] 2024-05-08T00:52:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11747], [object_number=14, tag_tid=1, entry_tid=11746]]], state=connected] 2024-05-08T00:52:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.status 2024-05-08T00:52:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:52:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_43 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_43 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_43.status 2024-05-08T00:52:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.status 2024-05-08T00:52:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-05-08T00:52:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.status 2024-05-08T00:52:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:52:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:52:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11747], [object_number=14, tag_tid=1, entry_tid=11746]]], state=connected] 2024-05-08T00:52:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=11747], [object_number=14, tag_tid=1, entry_tid=11746]]], state=connected] 2024-05-08T00:52:36.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.status 2024-05-08T00:52:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:52:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:52:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-05-08T00:52:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:52:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-05-08T00:52:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_43': 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aab564eb42 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aab564eb42 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:47:15 2024 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:47:15 2024 2024-05-08T00:52:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:48:16 2024 2024-05-08T00:52:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aab564eb42 2024-05-08T00:52:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: cb4e1076-0791-4a47-b9ba-f80603a3f80e 2024-05-08T00:52:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_43 snap 2024-05-08T00:52:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:52:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.export 2024-05-08T00:52:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_43.export 2024-05-08T00:52:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_43.export 2024-05-08T00:52:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.export 2024-05-08T00:52:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_43.export 2024-05-08T00:52:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_43.export 2024-05-08T00:52:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_43.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_43.export 2024-05-08T00:52:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_44 2024-05-08T00:52:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_44 snap 2024-05-08T00:52:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:52:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-05-08T00:52:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-05-08T00:52:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-05-08T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-05-08T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:52:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:52:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:52:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-05-08T00:52:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_44 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:52:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_44 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_44 ']' 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-05-08T00:52:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-05-08T00:52:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-05-08T00:52:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:52:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:52:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-05-08T00:52:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-05-08T00:52:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_44 2024-05-08T00:52:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-05-08T00:52:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_44 2024-05-08T00:52:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-05-08T00:52:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_44.status 2024-05-08T00:52:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.status 2024-05-08T00:52:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-05-08T00:52:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.status 2024-05-08T00:52:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:52:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:52:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.status 2024-05-08T00:52:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_44 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_44 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-05-08T00:52:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_44.status 2024-05-08T00:52:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.status 2024-05-08T00:52:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-05-08T00:52:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.status 2024-05-08T00:52:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:52:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:52:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:52:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.status 2024-05-08T00:52:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:52:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-05-08T00:52:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:52:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_44': 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10ad93617d2a 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10ad93617d2a 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:47:28 2024 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:47:28 2024 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:48:29 2024 2024-05-08T00:52:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10ad93617d2a 2024-05-08T00:52:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: be150828-976c-494e-b85d-30e721753423 2024-05-08T00:52:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_44 snap 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.export 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_44.export 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_44.export 2024-05-08T00:52:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.export 2024-05-08T00:52:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_44.export 2024-05-08T00:52:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_44.export 2024-05-08T00:52:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_44.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_44.export 2024-05-08T00:52:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_45 2024-05-08T00:52:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_45 snap 2024-05-08T00:52:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:52:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-05-08T00:52:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-05-08T00:52:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_45 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_45 2024-05-08T00:52:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:52:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_45 ']' 2024-05-08T00:52:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-05-08T00:52:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-05-08T00:52:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-05-08T00:52:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_45 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_45 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_45.status 2024-05-08T00:52:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.status 2024-05-08T00:52:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-05-08T00:52:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.status 2024-05-08T00:52:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:52:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:52:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12367], [object_number=14, tag_tid=1, entry_tid=12366]]], state=connected] 2024-05-08T00:52:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12367], [object_number=14, tag_tid=1, entry_tid=12366]]], state=connected] 2024-05-08T00:52:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.status 2024-05-08T00:52:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_45 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_45 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_45.status 2024-05-08T00:52:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.status 2024-05-08T00:52:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-05-08T00:52:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.status 2024-05-08T00:52:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:52:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:52:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12367], [object_number=14, tag_tid=1, entry_tid=12366]]], state=connected] 2024-05-08T00:52:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12367], [object_number=14, tag_tid=1, entry_tid=12366]]], state=connected] 2024-05-08T00:52:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.status 2024-05-08T00:52:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-05-08T00:52:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_45': 2024-05-08T00:52:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa77873ed3 2024-05-08T00:52:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa77873ed3 2024-05-08T00:52:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:47:34 2024 2024-05-08T00:52:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:47:34 2024 2024-05-08T00:52:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:48:35 2024 2024-05-08T00:52:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa77873ed3 2024-05-08T00:52:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: d30dd72f-1d0e-4c65-a931-767564d2930b 2024-05-08T00:52:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_45 snap 2024-05-08T00:52:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:52:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.export 2024-05-08T00:52:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_45.export 2024-05-08T00:52:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_45.export 2024-05-08T00:52:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.export 2024-05-08T00:52:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_45.export 2024-05-08T00:52:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_45.export 2024-05-08T00:52:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_45.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_45.export 2024-05-08T00:52:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_46 2024-05-08T00:52:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_46 snap 2024-05-08T00:52:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-05-08T00:52:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-05-08T00:52:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-05-08T00:52:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:52.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-05-08T00:52:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-05-08T00:52:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:52:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-05-08T00:52:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:52:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:52:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-05-08T00:52:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-05-08T00:52:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_46 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_46 2024-05-08T00:52:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_46 ']' 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-05-08T00:52:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-05-08T00:52:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:52:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-05-08T00:52:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_46.status 2024-05-08T00:52:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-05-08T00:52:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9523], [object_number=10, tag_tid=1, entry_tid=9522]]], state=connected] 2024-05-08T00:52:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=9524], [object_number=11, tag_tid=1, entry_tid=9523], [object_number=10, tag_tid=1, entry_tid=9522], [object_number=9, tag_tid=1, entry_tid=9521]]], state=connected] 2024-05-08T00:52:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-05-08T00:52:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:52:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-05-08T00:52:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:52:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-05-08T00:52:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_46.status 2024-05-08T00:52:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-05-08T00:52:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9523], [object_number=10, tag_tid=1, entry_tid=9522]]], state=connected] 2024-05-08T00:52:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=9524], [object_number=11, tag_tid=1, entry_tid=9523], [object_number=10, tag_tid=1, entry_tid=9522], [object_number=9, tag_tid=1, entry_tid=9521]]], state=connected] 2024-05-08T00:52:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=9524' 2024-05-08T00:52:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=9524' 2024-05-08T00:52:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=9524' '!=' '' 2024-05-08T00:52:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T00:52:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=9524' = '' 2024-05-08T00:52:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=9524' 2024-05-08T00:52:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:52:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_tag=2 2024-05-08T00:52:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=9524' 2024-05-08T00:52:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-05-08T00:52:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_tag=1 2024-05-08T00:52:53.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:53.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:52:53.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_entry=1 2024-05-08T00:52:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=9524' 2024-05-08T00:52:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-05-08T00:52:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut -d= -f 2 2024-05-08T00:52:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_entry=9524 2024-05-08T00:52:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 2 = 1 -a 1 -le 9524 2024-05-08T00:52:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_46 2024-05-08T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_46 ']' 2024-05-08T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-05-08T00:52:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-05-08T00:52:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-05-08T00:52:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-05-08T00:52:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:52:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:52:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:52:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:52:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:52:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:52:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-05-08T00:52:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_46.status 2024-05-08T00:52:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-05-08T00:52:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9523], [object_number=10, tag_tid=1, entry_tid=9522]]], state=connected] 2024-05-08T00:52:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9523], [object_number=10, tag_tid=1, entry_tid=9522]]], state=connected] 2024-05-08T00:52:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_46.status 2024-05-08T00:52:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-05-08T00:52:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 2 2024-05-08T00:52:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9523], [object_number=10, tag_tid=1, entry_tid=9522]]], state=connected] 2024-05-08T00:52:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9523], [object_number=10, tag_tid=1, entry_tid=9522]]], state=connected] 2024-05-08T00:52:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.status 2024-05-08T00:52:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-05-08T00:52:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_46': 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10adc3e0f555 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10adc3e0f555 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:47:46 2024 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:47:46 2024 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:48:47 2024 2024-05-08T00:52:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10adc3e0f555 2024-05-08T00:52:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: f3ead5c7-9c0c-4f2a-a3e4-dacba782313b 2024-05-08T00:52:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_46 snap 2024-05-08T00:52:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:52:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.export 2024-05-08T00:52:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_46.export 2024-05-08T00:52:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_46.export 2024-05-08T00:52:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.export 2024-05-08T00:52:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_46.export 2024-05-08T00:52:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_46.export 2024-05-08T00:52:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_46.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_46.export 2024-05-08T00:52:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_47 2024-05-08T00:52:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_47 snap 2024-05-08T00:52:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:52:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:52:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-05-08T00:52:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:52:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-05-08T00:52:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:52:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:52:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-05-08T00:52:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:52:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:52:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:52:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:52:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-05-08T00:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:52:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:52:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-05-08T00:52:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:52:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-05-08T00:52:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:52:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:52:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:52:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:52:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:52:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_47 2024-05-08T00:52:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:52:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:52:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:52:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:52:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:52:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:52:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_47 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_47 ']' 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:52:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:52:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:52:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:52:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:52:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:52:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:52:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:52:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:52:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:52:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:52:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:52:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:52:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:52:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:52:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:52:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_47 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_47 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-05-08T00:52:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_47.status 2024-05-08T00:52:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.status 2024-05-08T00:52:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-05-08T00:52:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.status 2024-05-08T00:52:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6835], [object_number=6, tag_tid=1, entry_tid=6834]]], state=connected] 2024-05-08T00:52:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6835], [object_number=6, tag_tid=1, entry_tid=6834]]], state=connected] 2024-05-08T00:52:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.status 2024-05-08T00:52:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-05-08T00:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-05-08T00:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_47 2024-05-08T00:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-05-08T00:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:52:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_47 2024-05-08T00:52:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:52:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-05-08T00:52:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_47.status 2024-05-08T00:52:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:52:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.status 2024-05-08T00:52:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-05-08T00:52:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.status 2024-05-08T00:52:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:52:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 2 2024-05-08T00:52:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:52:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6835], [object_number=6, tag_tid=1, entry_tid=6834]]], state=connected] 2024-05-08T00:52:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6835], [object_number=6, tag_tid=1, entry_tid=6834]]], state=connected] 2024-05-08T00:52:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.status 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:52:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:52:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-05-08T00:52:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_47': 2024-05-08T00:52:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:52:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:52:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa1ecaa1db 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa1ecaa1db 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:47:54 2024 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:47:54 2024 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:48:55 2024 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa1ecaa1db 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:52:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 3348438f-f39c-4fa7-abe2-1198b932d949 2024-05-08T00:52:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:52:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_47 snap 2024-05-08T00:52:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:52:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:52:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.export 2024-05-08T00:52:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_47.export 2024-05-08T00:52:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_47.export 2024-05-08T00:52:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.export 2024-05-08T00:52:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_47.export 2024-05-08T00:52:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:52:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_47.export 2024-05-08T00:52:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_47.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_47.export 2024-05-08T00:52:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:52:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:52:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_48 2024-05-08T00:52:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_48 snap 2024-05-08T00:52:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:52:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:52:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:52:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:52:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:53:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-05-08T00:53:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:53:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-05-08T00:53:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:53:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:53:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:53:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:53:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:53:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:53:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:53:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:53:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:53:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:53:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:53:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:53:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:53:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:53:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-05-08T00:53:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:53:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:53:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:53:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:53:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:53:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:53:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:53:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:53:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-05-08T00:53:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-05-08T00:53:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:53:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:53:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:53:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:53:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:53:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:53:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:53:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:53:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-05-08T00:53:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:53:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:53:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:53:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:53:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:53:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:53:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_48 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:53:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:53:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:53:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:53:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:53:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:53:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_48 2024-05-08T00:53:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:53:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:53:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_48 ']' 2024-05-08T00:53:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-05-08T00:53:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-05-08T00:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:53:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:53:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-05-08T00:53:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:53:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:53:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:53:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:53:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:53:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:53:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:53:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:53:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:53:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:53:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:53:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:53:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:53:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:53:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:53:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-05-08T00:53:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-05-08T00:53:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:53:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:53:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:53:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:53:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:53:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:53:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:53:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:53:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-05-08T00:53:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-05-08T00:53:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_48 2024-05-08T00:53:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-05-08T00:53:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_48 2024-05-08T00:53:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:53:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-05-08T00:53:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_48.status 2024-05-08T00:53:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:53:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.status 2024-05-08T00:53:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-05-08T00:53:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.status 2024-05-08T00:53:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:53:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:53:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:53:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:53:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:53:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.status 2024-05-08T00:53:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_48 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_48 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:53:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-05-08T00:53:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_48.status 2024-05-08T00:53:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:53:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.status 2024-05-08T00:53:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-05-08T00:53:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.status 2024-05-08T00:53:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:53:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:53:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:53:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:53:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-05-08T00:53:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.status 2024-05-08T00:53:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:53:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:53:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-05-08T00:53:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:53:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:53:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:53:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:53:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:53:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-05-08T00:53:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_48': 2024-05-08T00:53:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:53:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:53:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:53:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10adcafc1fa8 2024-05-08T00:53:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10adcafc1fa8 2024-05-08T00:53:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:53:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:53:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:53:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:53:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:48:02 2024 2024-05-08T00:53:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:48:02 2024 2024-05-08T00:53:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:49:03 2024 2024-05-08T00:53:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:53:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10adcafc1fa8 2024-05-08T00:53:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:53:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:53:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: c586e4bd-43ed-41b9-a24c-856a818e88ea 2024-05-08T00:53:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:53:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_48 snap 2024-05-08T00:53:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:53:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:53:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:53:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.export 2024-05-08T00:53:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_48.export 2024-05-08T00:53:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_48.export 2024-05-08T00:53:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.export 2024-05-08T00:53:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:53:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_48.export 2024-05-08T00:53:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:53:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_48.export 2024-05-08T00:53:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_48.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_48.export 2024-05-08T00:53:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_49 2024-05-08T00:53:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_49 snap 2024-05-08T00:53:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:53:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:53:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:53:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:53:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:53:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:53:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-05-08T00:53:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:53:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:53:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:53:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:53:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:53:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:53:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:53:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:53:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:53:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-05-08T00:53:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:53:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:53:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:53:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:53:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:53:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:53:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:53:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-05-08T00:53:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:53:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_49 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:53:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:53:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_49 2024-05-08T00:53:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:53:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_49 ']' 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:53:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:53:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-05-08T00:53:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:53:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:53:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:53:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:53:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:53:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:53:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-05-08T00:53:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-05-08T00:53:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_49 2024-05-08T00:53:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-05-08T00:53:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_49 2024-05-08T00:53:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:53:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-05-08T00:53:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_49.status 2024-05-08T00:53:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:53:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.status 2024-05-08T00:53:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-05-08T00:53:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.status 2024-05-08T00:53:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:53:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:53:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:53:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5318], [object_number=7, tag_tid=1, entry_tid=5315]]], state=connected] 2024-05-08T00:53:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5318], [object_number=7, tag_tid=1, entry_tid=5315]]], state=connected] 2024-05-08T00:53:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.status 2024-05-08T00:53:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:53:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-05-08T00:53:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_49 2024-05-08T00:53:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-05-08T00:53:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_49 2024-05-08T00:53:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:53:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-05-08T00:53:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_49.status 2024-05-08T00:53:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:53:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.status 2024-05-08T00:53:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-05-08T00:53:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.status 2024-05-08T00:53:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 1 2024-05-08T00:53:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 1 2024-05-08T00:53:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:53:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5318], [object_number=7, tag_tid=1, entry_tid=5315]]], state=connected] 2024-05-08T00:53:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=6, tag_tid=1, entry_tid=5318], [object_number=7, tag_tid=1, entry_tid=5315]]], state=connected] 2024-05-08T00:53:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.status 2024-05-08T00:53:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:53:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-05-08T00:53:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-05-08T00:53:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:53:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:53:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:53:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:53:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:53:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-05-08T00:53:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_49': 2024-05-08T00:53:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:53:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:53:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:53:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10aa31aeb12c 2024-05-08T00:53:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10aa31aeb12c 2024-05-08T00:53:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:53:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:48:09 2024 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:48:09 2024 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:49:10 2024 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10aa31aeb12c 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: cf0c34fd-0456-4e18-b8d9-dac46c7f7b7d 2024-05-08T00:53:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_49 snap 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.export 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_49.export 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_49.export 2024-05-08T00:53:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.export 2024-05-08T00:53:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:53:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_49.export 2024-05-08T00:53:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:53:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_49.export 2024-05-08T00:53:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_49.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_49.export 2024-05-08T00:53:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_50 2024-05-08T00:53:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_snap cluster2 mirror image_50 snap 2024-05-08T00:53:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:53:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:53:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-05-08T00:53:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-05-08T00:53:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:53:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-05-08T00:53:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:53:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=started 2024-05-08T00:53:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:53:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:53:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:53:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-05-08T00:53:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:53:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=started 2024-05-08T00:53:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local status_result 2024-05-08T00:53:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=stopped 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_instance=cluster1 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -i state 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance=cluster1 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local loop_instance 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sleep 0 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:53:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ seq 0 1 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:0 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:53:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:53:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T00:53:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T00:53:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:53:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-05-08T00:53:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-05-08T00:53:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror status : get status for rbd mirror 2024-05-08T00:53:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ shift 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:53:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local local_cluster=cluster1:1 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=local_cluster 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T00:53:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:53:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T00:53:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval local_cluster=cluster1 2024-05-08T00:53:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ local_cluster=cluster1 2024-05-08T00:53:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T00:53:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T00:53:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:53:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-05-08T00:53:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ return 0 2024-05-08T00:53:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ status_result=' "state": "Replaying"' 2024-05-08T00:53:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo ' "state": "Replaying"' 2024-05-08T00:53:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ grep -i Replaying 2024-05-08T00:53:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: "state": "Replaying" 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=started 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test started = started 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' journal = journal ']' 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_50 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local local_cluster=cluster1 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:53:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-05-08T00:53:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-05-08T00:53:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:53:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-08T00:53:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.2 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ flush cluster1 mirror image_50 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local 'cmd=rbd mirror flush' 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n image_50 ']' 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_instance=cluster1 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance=cluster1 2024-05-08T00:53:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loop_instance 2024-05-08T00:53:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-08T00:53:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-05-08T00:53:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:0 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T00:53:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:53:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:53:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T00:53:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T00:53:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T00:53:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T00:53:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-05-08T00:53:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-05-08T00:53:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd mirror flush : flush rbd mirror 2024-05-08T00:53:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:admin_socket: invalid command 2024-05-08T00:53:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T00:53:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-05-08T00:53:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T00:53:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:53:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1:1 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=1 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=1 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T00:53:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T00:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T00:53:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T00:53:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-05-08T00:53:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-05-08T00:53:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_50 2024-05-08T00:53:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-05-08T00:53:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_50 2024-05-08T00:53:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local id_regexp= 2024-05-08T00:53:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-05-08T00:53:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_50.status 2024-05-08T00:53:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:53:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.status 2024-05-08T00:53:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-05-08T00:53:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.status 2024-05-08T00:53:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:53:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:53:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:53:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12507], [object_number=14, tag_tid=1, entry_tid=12506]]], state=connected] 2024-05-08T00:53:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12507], [object_number=14, tag_tid=1, entry_tid=12506]]], state=connected] 2024-05-08T00:53:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.status 2024-05-08T00:53:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:53:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-05-08T00:53:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_50 2024-05-08T00:53:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-05-08T00:53:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T00:53:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:53:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_50 2024-05-08T00:53:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local 'id_regexp=..*' 2024-05-08T00:53:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-05-08T00:53:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo cluster2-mirror-image_50.status 2024-05-08T00:53:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ sed -e 's|[/ ]|_|g' 2024-05-08T00:53:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local status_log=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.status 2024-05-08T00:53:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-05-08T00:53:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ tee /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.status 2024-05-08T00:53:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:minimum_set: 3 2024-05-08T00:53:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:active_set: 3 2024-05-08T00:53:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:registered clients: 2024-05-08T00:53:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12507], [object_number=14, tag_tid=1, entry_tid=12506]]], state=connected] 2024-05-08T00:53:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: [id=af83744d-1cb7-4e10-9794-08f48dcc5545, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=15, tag_tid=1, entry_tid=12507], [object_number=14, tag_tid=1, entry_tid=12506]]], state=connected] 2024-05-08T00:53:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.status 2024-05-08T00:53:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:53:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-05-08T00:53:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-05-08T00:53:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:53:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:53:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:53:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:53:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-08T00:53:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T00:53:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-05-08T00:53:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_50': 2024-05-08T00:53:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 128 MiB in 32 objects 2024-05-08T00:53:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:53:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 1 2024-05-08T00:53:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 10adaeaf683f 2024-05-08T00:53:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.10adaeaf683f 2024-05-08T00:53:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:53:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:53:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:53:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:53:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:48:15 2024 2024-05-08T00:53:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:48:15 2024 2024-05-08T00:53:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:49:16 2024 2024-05-08T00:53:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: protected: False 2024-05-08T00:53:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 10adaeaf683f 2024-05-08T00:53:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:53:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:53:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: a19fcb1a-b27e-48c3-acbc-48d8aff7f9b2 2024-05-08T00:53:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: false 2024-05-08T00:53:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ compare_image_snaps mirror image_50 snap 2024-05-08T00:53:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:53:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap_name=snap 2024-05-08T00:53:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local ret=0 2024-05-08T00:53:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local rmt_export=/tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.export 2024-05-08T00:53:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local loc_export=/tmp/tmp.W1VfRby4SU/cluster1-mirror-image_50.export 2024-05-08T00:53:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_50.export 2024-05-08T00:53:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.export 2024-05-08T00:53:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:53:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_50.export 2024-05-08T00:53:14.603 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:14.613+0000 7f5309905640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-05-08T00:53:14.604 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:14.614+0000 7f5308903640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 4 2024-05-08T00:53:14.604 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:14.614+0000 7f53060fe640 -1 bdev(0x56208038ae00 /var/lib/ceph/osd/cluster1-0/block) aio_submit retries 3 2024-05-08T00:53:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-08T00:53:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cmp /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_50.export 2024-05-08T00:53:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /tmp/tmp.W1VfRby4SU/cluster2-mirror-image_50.export /tmp/tmp.W1VfRby4SU/cluster1-mirror-image_50.export 2024-05-08T00:53:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ testlog 'TEST: delete many images' 2024-05-08T00:53:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ tee -a /tmp/tmp.W1VfRby4SU/rbd-mirror.test.log 2024-05-08T00:53:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ date '+%F %T' 2024-05-08T00:53:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo 2024-05-08 00:53:14 TEST: delete many images 2024-05-08T00:53:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:2024-05-08 00:53:14 TEST: delete many images 2024-05-08T00:53:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 50 2024-05-08T00:53:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_1 2024-05-08T00:53:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-05-08T00:53:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:53:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-05-08T00:53:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_1 2024-05-08T00:53:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:53:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_1 2024-05-08T00:53:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:53:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-05-08T00:53:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-05-08T00:53:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_2 2024-05-08T00:53:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-05-08T00:53:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:53:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-05-08T00:53:16.374 INFO:tasks.ceph.cluster1.osd.2.smithi064.stderr:2024-05-08T00:53:16.384+0000 7fbb0a622640 -1 reset not still connected to 0x55a780d23ee0 2024-05-08T00:53:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_2 2024-05-08T00:53:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:53:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_2 2024-05-08T00:53:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:53:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-05-08T00:53:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-05-08T00:53:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_3 2024-05-08T00:53:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-05-08T00:53:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:53:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-05-08T00:53:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_3 2024-05-08T00:53:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:53:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_3 2024-05-08T00:53:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:53:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-05-08T00:53:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-05-08T00:53:17.683 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.690+0000 7f5317d5e640 -1 reset not still connected to 0x562092abe8f0 2024-05-08T00:53:17.683 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.690+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:53:17.683 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.690+0000 7f5317d5e640 -1 reset not still connected to 0x56209393a820 2024-05-08T00:53:17.686 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.697+0000 7f5317d5e640 -1 reset not still connected to 0x562099e6ad00 2024-05-08T00:53:17.691 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.701+0000 7f5317d5e640 -1 reset not still connected to 0x56209ddcc680 2024-05-08T00:53:17.691 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.701+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:53:17.691 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.701+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:53:17.691 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.701+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:53:17.694 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.702+0000 7f5317d5e640 -1 reset not still connected to 0x5620a57cc750 2024-05-08T00:53:17.694 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.702+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:53:17.694 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.702+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:53:17.694 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.702+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:53:17.694 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.702+0000 7f5317d5e640 -1 reset not still connected to 0x5620b325cc30 2024-05-08T00:53:17.694 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.702+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:53:17.694 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.702+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6ca1ba0 2024-05-08T00:53:17.695 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.702+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:53:17.695 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:17.702+0000 7f5317d5e640 -1 reset not still connected to 0x5620b8f7d790 2024-05-08T00:53:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_4 2024-05-08T00:53:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-05-08T00:53:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:53:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-05-08T00:53:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_4 2024-05-08T00:53:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:53:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_4 2024-05-08T00:53:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:53:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-05-08T00:53:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-05-08T00:53:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_5 2024-05-08T00:53:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-05-08T00:53:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:53:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-05-08T00:53:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_5 2024-05-08T00:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_5 2024-05-08T00:53:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:53:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-05-08T00:53:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-05-08T00:53:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_6 2024-05-08T00:53:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-05-08T00:53:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:53:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-05-08T00:53:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_6 2024-05-08T00:53:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:53:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_6 2024-05-08T00:53:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:53:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-05-08T00:53:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-05-08T00:53:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_7 2024-05-08T00:53:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-05-08T00:53:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:53:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-05-08T00:53:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_7 2024-05-08T00:53:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:53:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_7 2024-05-08T00:53:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:53:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-05-08T00:53:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-05-08T00:53:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_8 2024-05-08T00:53:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-05-08T00:53:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:53:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-05-08T00:53:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_8 2024-05-08T00:53:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:53:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_8 2024-05-08T00:53:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:53:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-05-08T00:53:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-05-08T00:53:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_9 2024-05-08T00:53:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-05-08T00:53:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:53:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-05-08T00:53:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_9 2024-05-08T00:53:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:53:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_9 2024-05-08T00:53:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:53:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-05-08T00:53:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-05-08T00:53:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_10 2024-05-08T00:53:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-05-08T00:53:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:53:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-05-08T00:53:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_10 2024-05-08T00:53:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:53:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_10 2024-05-08T00:53:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:53:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-05-08T00:53:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-05-08T00:53:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_11 2024-05-08T00:53:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-05-08T00:53:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:53:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-05-08T00:53:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_11 2024-05-08T00:53:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:53:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_11 2024-05-08T00:53:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:53:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-05-08T00:53:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-05-08T00:53:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_12 2024-05-08T00:53:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-05-08T00:53:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:53:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-05-08T00:53:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_12 2024-05-08T00:53:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:53:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_12 2024-05-08T00:53:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:53:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-05-08T00:53:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-05-08T00:53:26.722 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.733+0000 7f5317d5e640 -1 reset not still connected to 0x56208cfbdd40 2024-05-08T00:53:26.722 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.733+0000 7f5317d5e640 -1 reset not still connected to 0x562092abe8f0 2024-05-08T00:53:26.723 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.733+0000 7f5317d5e640 -1 reset not still connected to 0x562093912750 2024-05-08T00:53:26.723 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.733+0000 7f5317d5e640 -1 reset not still connected to 0x56209393a820 2024-05-08T00:53:26.723 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.733+0000 7f5317d5e640 -1 reset not still connected to 0x56209ddcc680 2024-05-08T00:53:26.723 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.733+0000 7f5317d5e640 -1 reset not still connected to 0x5620a06a3450 2024-05-08T00:53:26.723 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.733+0000 7f5317d5e640 -1 reset not still connected to 0x5620a0eff6c0 2024-05-08T00:53:26.723 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.733+0000 7f5317d5e640 -1 reset not still connected to 0x5620a2ceb6c0 2024-05-08T00:53:26.723 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.733+0000 7f5317d5e640 -1 reset not still connected to 0x5620a712f520 2024-05-08T00:53:26.724 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.735+0000 7f5317d5e640 -1 reset not still connected to 0x5620b0f4d380 2024-05-08T00:53:26.724 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.735+0000 7f5317d5e640 -1 reset not still connected to 0x5620b3169110 2024-05-08T00:53:26.724 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.735+0000 7f5317d5e640 -1 reset not still connected to 0x5620b6c3e1a0 2024-05-08T00:53:26.724 INFO:tasks.ceph.cluster1.osd.0.smithi064.stderr:2024-05-08T00:53:26.735+0000 7f5317d5e640 -1 reset not still connected to 0x5620b72f04e0 2024-05-08T00:53:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_13 2024-05-08T00:53:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-05-08T00:53:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:53:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-05-08T00:53:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_13 2024-05-08T00:53:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:53:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_13 2024-05-08T00:53:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:53:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-05-08T00:53:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-05-08T00:53:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_14 2024-05-08T00:53:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-05-08T00:53:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:53:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-05-08T00:53:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_14 2024-05-08T00:53:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:53:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_14 2024-05-08T00:53:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:53:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-05-08T00:53:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-05-08T00:53:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_15 2024-05-08T00:53:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-05-08T00:53:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:53:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-05-08T00:53:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_15 2024-05-08T00:53:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:53:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_15 2024-05-08T00:53:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:53:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-05-08T00:53:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-05-08T00:53:29.739 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:53:29.750+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd6680 2024-05-08T00:53:29.739 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:53:29.750+0000 7f2ee8664640 -1 reset not still connected to 0x563243dd68f0 2024-05-08T00:53:29.740 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:53:29.750+0000 7f2ee8664640 -1 reset not still connected to 0x563245aab2b0 2024-05-08T00:53:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_16 2024-05-08T00:53:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-05-08T00:53:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:53:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-05-08T00:53:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_16 2024-05-08T00:53:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:53:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_16 2024-05-08T00:53:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:53:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-05-08T00:53:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-05-08T00:53:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_17 2024-05-08T00:53:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-05-08T00:53:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:53:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-05-08T00:53:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_17 2024-05-08T00:53:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:53:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_17 2024-05-08T00:53:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:53:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-05-08T00:53:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-05-08T00:53:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_18 2024-05-08T00:53:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-05-08T00:53:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:53:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-05-08T00:53:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_18 2024-05-08T00:53:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:53:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_18 2024-05-08T00:53:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:53:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-05-08T00:53:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-05-08T00:53:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_19 2024-05-08T00:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-05-08T00:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-05-08T00:53:33.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_19 2024-05-08T00:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_19 2024-05-08T00:53:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:53:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-05-08T00:53:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-05-08T00:53:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_20 2024-05-08T00:53:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-05-08T00:53:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:53:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-05-08T00:53:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_20 2024-05-08T00:53:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:53:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_20 2024-05-08T00:53:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:53:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-05-08T00:53:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-05-08T00:53:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_21 2024-05-08T00:53:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-05-08T00:53:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:53:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-05-08T00:53:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_21 2024-05-08T00:53:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:53:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_21 2024-05-08T00:53:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:53:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-05-08T00:53:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-05-08T00:53:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_22 2024-05-08T00:53:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-05-08T00:53:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:53:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-05-08T00:53:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_22 2024-05-08T00:53:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:53:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_22 2024-05-08T00:53:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:53:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-05-08T00:53:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-05-08T00:53:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_23 2024-05-08T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-05-08T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-05-08T00:53:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_23 2024-05-08T00:53:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:53:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_23 2024-05-08T00:53:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:53:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-05-08T00:53:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-05-08T00:53:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_24 2024-05-08T00:53:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-05-08T00:53:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:53:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-05-08T00:53:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_24 2024-05-08T00:53:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:53:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_24 2024-05-08T00:53:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:53:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-05-08T00:53:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-05-08T00:53:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_25 2024-05-08T00:53:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-05-08T00:53:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:53:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-05-08T00:53:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_25 2024-05-08T00:53:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:53:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_25 2024-05-08T00:53:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:53:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-05-08T00:53:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-05-08T00:53:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_26 2024-05-08T00:53:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-05-08T00:53:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:53:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-05-08T00:53:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_26 2024-05-08T00:53:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:53:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_26 2024-05-08T00:53:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:53:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-05-08T00:53:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-05-08T00:53:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_27 2024-05-08T00:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-05-08T00:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-05-08T00:53:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_27 2024-05-08T00:53:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:53:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_27 2024-05-08T00:53:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:53:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-05-08T00:53:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-05-08T00:53:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_28 2024-05-08T00:53:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-05-08T00:53:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:53:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-05-08T00:53:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_28 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_28 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:53:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-05-08T00:53:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-05-08T00:53:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_29 2024-05-08T00:53:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-05-08T00:53:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:53:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-05-08T00:53:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_29 2024-05-08T00:53:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:53:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_29 2024-05-08T00:53:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:53:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-05-08T00:53:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-05-08T00:53:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_30 2024-05-08T00:53:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-05-08T00:53:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:53:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-05-08T00:53:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_30 2024-05-08T00:53:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:53:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_30 2024-05-08T00:53:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:53:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-05-08T00:53:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-05-08T00:53:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_31 2024-05-08T00:53:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-05-08T00:53:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:53:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-05-08T00:53:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_31 2024-05-08T00:53:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:53:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_31 2024-05-08T00:53:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:53:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-05-08T00:53:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-05-08T00:53:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_32 2024-05-08T00:53:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-05-08T00:53:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:53:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-05-08T00:53:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_32 2024-05-08T00:53:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:53:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_32 2024-05-08T00:53:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:53:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-05-08T00:53:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-05-08T00:53:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_33 2024-05-08T00:53:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-05-08T00:53:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:53:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-05-08T00:53:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_33 2024-05-08T00:53:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:53:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_33 2024-05-08T00:53:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:53:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-05-08T00:53:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-05-08T00:53:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_34 2024-05-08T00:53:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-05-08T00:53:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:53:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-05-08T00:53:47.889 INFO:tasks.ceph.cluster1.osd.1.smithi064.stderr:2024-05-08T00:53:47.900+0000 7f2ee8664640 -1 reset not still connected to 0x563243799ad0 2024-05-08T00:53:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_34 2024-05-08T00:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_34 2024-05-08T00:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-05-08T00:53:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-05-08T00:53:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_35 2024-05-08T00:53:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-05-08T00:53:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:53:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-05-08T00:53:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_35 2024-05-08T00:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_35 2024-05-08T00:53:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:53:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-05-08T00:53:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-05-08T00:53:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_36 2024-05-08T00:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-05-08T00:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-05-08T00:53:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_36 2024-05-08T00:53:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:53:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_36 2024-05-08T00:53:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:53:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-05-08T00:53:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-05-08T00:53:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_37 2024-05-08T00:53:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-05-08T00:53:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:53:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-05-08T00:53:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_37 2024-05-08T00:53:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:53:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_37 2024-05-08T00:53:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:53:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-05-08T00:53:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-05-08T00:53:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_38 2024-05-08T00:53:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-05-08T00:53:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:53:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-05-08T00:53:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_38 2024-05-08T00:53:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:53:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_38 2024-05-08T00:53:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:53:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-05-08T00:53:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-05-08T00:53:52.806 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:53:52.817+0000 7fa00a173640 -1 reset not still connected to 0x5564f4c3bba0 2024-05-08T00:53:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_39 2024-05-08T00:53:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-05-08T00:53:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:53:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-05-08T00:53:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_39 2024-05-08T00:53:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:53:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_39 2024-05-08T00:53:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:53:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-05-08T00:53:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-05-08T00:53:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_40 2024-05-08T00:53:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-05-08T00:53:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:53:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-05-08T00:53:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_40 2024-05-08T00:53:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:53:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_40 2024-05-08T00:53:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:53:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-05-08T00:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-05-08T00:53:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_41 2024-05-08T00:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-05-08T00:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:53:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-05-08T00:53:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_41 2024-05-08T00:53:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:53:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_41 2024-05-08T00:53:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:53:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-05-08T00:53:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-05-08T00:53:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_42 2024-05-08T00:53:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-05-08T00:53:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:53:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-05-08T00:53:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_42 2024-05-08T00:53:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:53:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_42 2024-05-08T00:53:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:53:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-05-08T00:53:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-05-08T00:53:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_43 2024-05-08T00:53:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-05-08T00:53:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:53:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-05-08T00:53:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_43 2024-05-08T00:53:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:53:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_43 2024-05-08T00:53:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:53:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-05-08T00:53:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-05-08T00:53:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_44 2024-05-08T00:53:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-05-08T00:53:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:53:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-05-08T00:53:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_44 2024-05-08T00:53:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:53:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_44 2024-05-08T00:53:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:53:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-05-08T00:53:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-05-08T00:53:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_45 2024-05-08T00:53:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-05-08T00:53:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:53:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-05-08T00:53:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:53:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_45 2024-05-08T00:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_45 2024-05-08T00:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-05-08T00:53:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-05-08T00:53:59.862 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:53:59.873+0000 7fa00a173640 -1 reset not still connected to 0x5564ee270270 2024-05-08T00:53:59.862 INFO:tasks.ceph.cluster2.osd.2.smithi116.stderr:2024-05-08T00:53:59.873+0000 7fa00a173640 -1 reset not still connected to 0x55650e944820 2024-05-08T00:53:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:53:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:53:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:53:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_46 2024-05-08T00:53:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-05-08T00:53:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:53:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:53:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:53:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:53:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-05-08T00:54:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:54:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_46 2024-05-08T00:54:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:54:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:54:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:54:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_46 2024-05-08T00:54:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:54:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-05-08T00:54:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-05-08T00:54:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:54:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_47 2024-05-08T00:54:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-05-08T00:54:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:54:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:54:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-05-08T00:54:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:54:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_47 2024-05-08T00:54:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:54:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:54:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:54:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_47 2024-05-08T00:54:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:54:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-05-08T00:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-05-08T00:54:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:54:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_48 2024-05-08T00:54:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-05-08T00:54:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:54:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:54:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-05-08T00:54:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_48 2024-05-08T00:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_48 2024-05-08T00:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-05-08T00:54:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-05-08T00:54:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:54:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_49 2024-05-08T00:54:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-05-08T00:54:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:54:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:54:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-05-08T00:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:54:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_49 2024-05-08T00:54:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:54:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:54:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:54:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_49 2024-05-08T00:54:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:54:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-05-08T00:54:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-05-08T00:54:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:54:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_50 2024-05-08T00:54:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-05-08T00:54:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:54:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local snap=snap 2024-05-08T00:54:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-05-08T00:54:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing snap: 100% complete...done. 2024-05-08T00:54:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_50 2024-05-08T00:54:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:54:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:54:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:54:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_50 2024-05-08T00:54:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:54:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-05-08T00:54:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-05-08T00:54:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-05-08T00:54:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ testlog 'TEST: image deletions should propagate' 2024-05-08T00:54:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ tee -a /tmp/tmp.W1VfRby4SU/rbd-mirror.test.log 2024-05-08T00:54:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ date '+%F %T' 2024-05-08T00:54:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo 2024-05-08 00:54:05 TEST: image deletions should propagate 2024-05-08T00:54:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:2024-05-08 00:54:05 TEST: image deletions should propagate 2024-05-08T00:54:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-05-08T00:54:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_count=0 2024-05-08T00:54:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local count 2024-05-08T00:54:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local last_count=0 2024-05-08T00:54:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ true 2024-05-08T00:54:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 40 2024-05-08T00:54:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in `seq 1 40` 2024-05-08T00:54:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 1 -ne 1 2024-05-08T00:54:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-05-08T00:54:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep 'images: ' 2024-05-08T00:54:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut '-d ' -f 2 2024-05-08T00:54:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: failed to query services: (13) Permission denied 2024-05-08T00:54:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ count=0 2024-05-08T00:54:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 = 0 2024-05-08T00:54:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-05-08T00:54:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state 2024-05-08T00:54:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 40 2024-05-08T00:54:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in `seq 1 40` 2024-05-08T00:54:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 1 -ne 1 2024-05-08T00:54:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-05-08T00:54:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep 'image health:' 2024-05-08T00:54:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cut '-d ' -f 3 2024-05-08T00:54:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: failed to query services: (13) Permission denied 2024-05-08T00:54:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ state=OK 2024-05-08T00:54:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test OK = ERROR 2024-05-08T00:54:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test OK = OK 2024-05-08T00:54:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 50 2024-05-08T00:54:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_1 2024-05-08T00:54:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-05-08T00:54:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_1 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-05-08T00:54:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-05-08T00:54:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_1 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:54:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-05-08T00:54:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-05-08T00:54:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_2 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_2 2024-05-08T00:54:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:54:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-05-08T00:54:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-05-08T00:54:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_2 2024-05-08T00:54:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:54:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-05-08T00:54:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-05-08T00:54:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_3 2024-05-08T00:54:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-05-08T00:54:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:54:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_3 2024-05-08T00:54:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:54:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-05-08T00:54:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-05-08T00:54:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-05-08T00:54:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:54:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_3 2024-05-08T00:54:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:54:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-05-08T00:54:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-05-08T00:54:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_4 2024-05-08T00:54:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-05-08T00:54:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_4 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:54:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-05-08T00:54:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-05-08T00:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-05-08T00:54:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_4 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:54:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-05-08T00:54:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-05-08T00:54:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_5 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_5 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-05-08T00:54:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-05-08T00:54:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-05-08T00:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:54:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_5 2024-05-08T00:54:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:54:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-05-08T00:54:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-05-08T00:54:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_6 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_6 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:54:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-05-08T00:54:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-05-08T00:54:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-05-08T00:54:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:54:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_6 2024-05-08T00:54:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:54:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-05-08T00:54:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-05-08T00:54:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_7 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_7 2024-05-08T00:54:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:54:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-05-08T00:54:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-05-08T00:54:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-05-08T00:54:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:54:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_7 2024-05-08T00:54:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:54:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-05-08T00:54:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_8 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_8 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-05-08T00:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-05-08T00:54:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-05-08T00:54:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:54:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_8 2024-05-08T00:54:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:54:06.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-05-08T00:54:06.551 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-05-08T00:54:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_9 2024-05-08T00:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-05-08T00:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_9 2024-05-08T00:54:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T00:54:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-05-08T00:54:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-05-08T00:54:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-05-08T00:54:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_9 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-05-08T00:54:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-05-08T00:54:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_10 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_10 2024-05-08T00:54:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T00:54:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-05-08T00:54:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-05-08T00:54:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-05-08T00:54:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T00:54:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_10 2024-05-08T00:54:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T00:54:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-05-08T00:54:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-05-08T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_11 2024-05-08T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-05-08T00:54:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_11 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-05-08T00:54:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-05-08T00:54:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-05-08T00:54:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_11 2024-05-08T00:54:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_11 2024-05-08T00:54:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_11 2024-05-08T00:54:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-05-08T00:54:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_12 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:54:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_12 2024-05-08T00:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-05-08T00:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-05-08T00:54:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-05-08T00:54:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_12 2024-05-08T00:54:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_12 2024-05-08T00:54:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_12 2024-05-08T00:54:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-05-08T00:54:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-05-08T00:54:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_13 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:54:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_13 2024-05-08T00:54:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:54:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-05-08T00:54:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-05-08T00:54:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-05-08T00:54:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_13 2024-05-08T00:54:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_13 2024-05-08T00:54:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_13 2024-05-08T00:54:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-05-08T00:54:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_14 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:54:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_14 2024-05-08T00:54:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_14 2024-05-08T00:54:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-05-08T00:54:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-05-08T00:54:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-05-08T00:54:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_14 2024-05-08T00:54:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_14 2024-05-08T00:54:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_14 2024-05-08T00:54:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-05-08T00:54:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_15 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_15 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_15 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-05-08T00:54:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-05-08T00:54:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-05-08T00:54:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_15 2024-05-08T00:54:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_15 2024-05-08T00:54:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_15 2024-05-08T00:54:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-05-08T00:54:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-05-08T00:54:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_16 2024-05-08T00:54:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_16 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_16 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-05-08T00:54:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-05-08T00:54:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-05-08T00:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_16 2024-05-08T00:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_16 2024-05-08T00:54:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_16 2024-05-08T00:54:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-05-08T00:54:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-05-08T00:54:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_17 2024-05-08T00:54:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-05-08T00:54:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_17 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_17 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-05-08T00:54:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-05-08T00:54:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_17 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_17 2024-05-08T00:54:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_17 2024-05-08T00:54:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-05-08T00:54:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-05-08T00:54:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_18 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_18 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_18 2024-05-08T00:54:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-05-08T00:54:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-05-08T00:54:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-05-08T00:54:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_18 2024-05-08T00:54:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_18 2024-05-08T00:54:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_18 2024-05-08T00:54:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-05-08T00:54:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-05-08T00:54:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_19 2024-05-08T00:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-05-08T00:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_19 2024-05-08T00:54:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_19 2024-05-08T00:54:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-05-08T00:54:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-05-08T00:54:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-05-08T00:54:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_19 2024-05-08T00:54:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_19 2024-05-08T00:54:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_19 2024-05-08T00:54:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-05-08T00:54:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-05-08T00:54:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_20 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_20 2024-05-08T00:54:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_20 2024-05-08T00:54:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-05-08T00:54:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-05-08T00:54:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-05-08T00:54:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_20 2024-05-08T00:54:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_20 2024-05-08T00:54:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_20 2024-05-08T00:54:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-05-08T00:54:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-05-08T00:54:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_21 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_21 2024-05-08T00:54:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_21 2024-05-08T00:54:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-05-08T00:54:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-05-08T00:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_21 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_21 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_21 2024-05-08T00:54:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-05-08T00:54:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_22 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_22 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_22 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-05-08T00:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-05-08T00:54:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-05-08T00:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_22 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_22 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_22 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-05-08T00:54:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-05-08T00:54:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_23 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_23 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_23 2024-05-08T00:54:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-05-08T00:54:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-05-08T00:54:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_23 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_23 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_23 2024-05-08T00:54:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-05-08T00:54:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-05-08T00:54:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_24 2024-05-08T00:54:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_24 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_24 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-05-08T00:54:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-05-08T00:54:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_24 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_24 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_24 2024-05-08T00:54:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-05-08T00:54:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-05-08T00:54:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_25 2024-05-08T00:54:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_25 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_25 2024-05-08T00:54:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-05-08T00:54:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-05-08T00:54:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-05-08T00:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_25 2024-05-08T00:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_25 2024-05-08T00:54:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_25 2024-05-08T00:54:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-05-08T00:54:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-05-08T00:54:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_26 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_26 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_26 2024-05-08T00:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-05-08T00:54:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-05-08T00:54:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-05-08T00:54:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_26 2024-05-08T00:54:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_26 2024-05-08T00:54:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_26 2024-05-08T00:54:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-05-08T00:54:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_27 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_27 2024-05-08T00:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_27 2024-05-08T00:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-05-08T00:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-05-08T00:54:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-05-08T00:54:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_27 2024-05-08T00:54:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_27 2024-05-08T00:54:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_27 2024-05-08T00:54:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-05-08T00:54:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-05-08T00:54:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_28 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_28 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_28 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-05-08T00:54:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-05-08T00:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-05-08T00:54:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_28 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_28 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_28 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-05-08T00:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-05-08T00:54:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_29 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_29 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_29 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-05-08T00:54:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-05-08T00:54:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-05-08T00:54:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_29 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_29 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_29 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-05-08T00:54:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-05-08T00:54:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_30 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_30 2024-05-08T00:54:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_30 2024-05-08T00:54:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-05-08T00:54:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-05-08T00:54:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-05-08T00:54:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_30 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_30 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_30 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-05-08T00:54:10.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-05-08T00:54:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_31 2024-05-08T00:54:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-05-08T00:54:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_31 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_31 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-05-08T00:54:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-05-08T00:54:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-05-08T00:54:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_31 2024-05-08T00:54:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_31 2024-05-08T00:54:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_31 2024-05-08T00:54:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-05-08T00:54:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-05-08T00:54:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:11.205 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_32 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_32 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_32 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-05-08T00:54:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-05-08T00:54:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_32 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_32 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_32 2024-05-08T00:54:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-05-08T00:54:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_33 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:54:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_33 2024-05-08T00:54:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_33 2024-05-08T00:54:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-05-08T00:54:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-05-08T00:54:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-05-08T00:54:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_33 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_33 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_33 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-05-08T00:54:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-05-08T00:54:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_34 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_34 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_34 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-05-08T00:54:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-05-08T00:54:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-05-08T00:54:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_34 2024-05-08T00:54:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_34 2024-05-08T00:54:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_34 2024-05-08T00:54:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-05-08T00:54:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_35 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_35 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-05-08T00:54:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-05-08T00:54:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-05-08T00:54:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_35 2024-05-08T00:54:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_35 2024-05-08T00:54:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_35 2024-05-08T00:54:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-05-08T00:54:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-05-08T00:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_36 2024-05-08T00:54:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_36 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_36 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-05-08T00:54:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-05-08T00:54:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-05-08T00:54:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_36 2024-05-08T00:54:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_36 2024-05-08T00:54:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_36 2024-05-08T00:54:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-05-08T00:54:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_37 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_37 2024-05-08T00:54:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:54:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-05-08T00:54:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-05-08T00:54:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-05-08T00:54:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_37 2024-05-08T00:54:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_37 2024-05-08T00:54:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_37 2024-05-08T00:54:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-05-08T00:54:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-05-08T00:54:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_38 2024-05-08T00:54:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-05-08T00:54:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:54:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_38 2024-05-08T00:54:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_38 2024-05-08T00:54:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-05-08T00:54:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-05-08T00:54:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-05-08T00:54:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_38 2024-05-08T00:54:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_38 2024-05-08T00:54:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_38 2024-05-08T00:54:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-05-08T00:54:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-05-08T00:54:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_39 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_39 2024-05-08T00:54:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:54:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-05-08T00:54:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-05-08T00:54:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_39 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_39 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_39 2024-05-08T00:54:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-05-08T00:54:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-05-08T00:54:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_40 2024-05-08T00:54:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-05-08T00:54:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_40 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_40 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-05-08T00:54:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-05-08T00:54:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-05-08T00:54:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_40 2024-05-08T00:54:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_40 2024-05-08T00:54:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_40 2024-05-08T00:54:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-05-08T00:54:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-05-08T00:54:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_41 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_41 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_41 2024-05-08T00:54:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-05-08T00:54:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-05-08T00:54:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-05-08T00:54:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_41 2024-05-08T00:54:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_41 2024-05-08T00:54:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_41 2024-05-08T00:54:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-05-08T00:54:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-05-08T00:54:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_42 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_42 2024-05-08T00:54:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_42 2024-05-08T00:54:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-05-08T00:54:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-05-08T00:54:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-05-08T00:54:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_42 2024-05-08T00:54:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_42 2024-05-08T00:54:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_42 2024-05-08T00:54:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-05-08T00:54:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-05-08T00:54:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_43 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_43 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_43 2024-05-08T00:54:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-05-08T00:54:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-05-08T00:54:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:13.173 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-05-08T00:54:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_43 2024-05-08T00:54:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_43 2024-05-08T00:54:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_43 2024-05-08T00:54:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-05-08T00:54:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_44 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:54:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_44 2024-05-08T00:54:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_44 2024-05-08T00:54:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-05-08T00:54:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-05-08T00:54:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-05-08T00:54:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_44 2024-05-08T00:54:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_44 2024-05-08T00:54:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_44 2024-05-08T00:54:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-05-08T00:54:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-05-08T00:54:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_45 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_45 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_45 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-05-08T00:54:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-05-08T00:54:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_45 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_45 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_45 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-05-08T00:54:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-05-08T00:54:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_46 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_46 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-05-08T00:54:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-05-08T00:54:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-05-08T00:54:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_46 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_46 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_46 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-05-08T00:54:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_47 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_47 2024-05-08T00:54:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_47 2024-05-08T00:54:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-05-08T00:54:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-05-08T00:54:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-05-08T00:54:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_47 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_47 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_47 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-05-08T00:54:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-05-08T00:54:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_48 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_48 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_48 2024-05-08T00:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-05-08T00:54:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-05-08T00:54:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_48 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_48 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_48 2024-05-08T00:54:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-05-08T00:54:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_49 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_49 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_49 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-05-08T00:54:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-05-08T00:54:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-05-08T00:54:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_49 2024-05-08T00:54:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_49 2024-05-08T00:54:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_49 2024-05-08T00:54:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-05-08T00:54:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-05-08T00:54:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_50 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_50 2024-05-08T00:54:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_50 2024-05-08T00:54:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-05-08T00:54:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-05-08T00:54:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:54:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-05-08T00:54:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_50 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_50 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_50 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-05-08T00:54:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-05-08T00:54:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-05-08T00:54:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ tee -a /tmp/tmp.W1VfRby4SU/rbd-mirror.test.log 2024-05-08T00:54:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ date '+%F %T' 2024-05-08T00:54:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo 2024-05-08 00:54:14 TEST: delete images during bootstrap 2024-05-08T00:54:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:2024-05-08 00:54:14 TEST: delete images during bootstrap 2024-05-08T00:54:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-05-08T00:54:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=image 2024-05-08T00:54:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-05-08T00:54:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:note: changing mirroring mode from pool to image 2024-05-08T00:54:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-05-08T00:54:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=image 2024-05-08T00:54:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-05-08T00:54:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:note: changing mirroring mode from pool to image 2024-05-08T00:54:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ start_mirror cluster1 2024-05-08T00:54:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set_cluster_instance cluster1 cluster instance 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local val=cluster1 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster_var_name=cluster 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local instance_var_name=instance 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cluster=cluster1 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=cluster1 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' cluster1 = cluster1 ']' 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ instance=0 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval cluster=cluster1 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ eval instance=0 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T00:54:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-05-08T00:54:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:2024-05-08T00:54:14.901+0000 7f0a9fcee640 -1 open: failed to lock pidfile /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid because another process locked it': (11) Resource temporarily unavailable 2024-05-08T00:54:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=test 2024-05-08T00:54:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 1 10 2024-05-08T00:54:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_1 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_1 512M 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:54:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-05-08T00:54:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_1 2024-05-08T00:54:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:54:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 journal 2024-05-08T00:54:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T00:54:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-05-08T00:54:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_1': 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1c7bed25a4a3 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1c7bed25a4a3 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:54:14 2024 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:54:14 2024 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:54:14 2024 2024-05-08T00:54:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1c7bed25a4a3 2024-05-08T00:54:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:54:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:54:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: ce2a9dd4-c2f4-4fc9-a097-d7d98ac92d6e 2024-05-08T00:54:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T00:54:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_1 2024-05-08T00:54:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:54:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:54:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:54:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:54:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 40278016 bytes 2024-05-08T00:54:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 68142080 bytes 2024-05-08T00:54:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 122178560 bytes 2024-05-08T00:54:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 144153600 bytes 2024-05-08T00:54:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 163018752 bytes 2024-05-08T00:54:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:54:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:54:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:54:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-05-08T00:54:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T00:54:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_1 2024-05-08T00:54:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:54:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-05-08T00:54:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aa62c98d16 2024-05-08T00:54:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_1 present 10aa62c98d16 2024-05-08T00:54:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T00:54:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aa62c98d16 2024-05-08T00:54:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_1 2024-05-08T00:54:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:54:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-05-08T00:54:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10aa62c98d16 2024-05-08T00:54:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10aa62c98d16 2024-05-08T00:54:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10aa62c98d16 -o 10aa62c98d16 = 10aa62c98d16 2024-05-08T00:54:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:54:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T00:54:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_1 2024-05-08T00:54:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-05-08T00:54:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-05-08T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_1 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-05-08T00:54:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aa62c98d16 2024-05-08T00:54:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:54:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:54:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_1 deleted 10aa62c98d16 2024-05-08T00:54:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aa62c98d16 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_1 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_1 2024-05-08T00:54:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-05-08T00:54:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-05-08T00:54:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:54:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:54:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:54:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_1 2024-05-08T00:54:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-05-08T00:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_1 2024-05-08T00:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:54:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_1 2024-05-08T00:54:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_1 2024-05-08T00:54:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-05-08T00:54:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-05-08T00:54:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_2 2024-05-08T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_2 512M 2024-05-08T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:54:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:54:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:54:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:54:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:54:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:54:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:54:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-05-08T00:54:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_2 2024-05-08T00:54:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:54:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:54:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 journal 2024-05-08T00:54:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T00:54:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-05-08T00:54:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_2': 2024-05-08T00:54:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:54:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:54:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T00:54:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1c93f01e20dd 2024-05-08T00:54:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1c93f01e20dd 2024-05-08T00:54:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:54:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:54:56 2024 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:54:56 2024 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:54:56 2024 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1c93f01e20dd 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 4eb870f5-31f0-49bc-a567-87a8bdbfac50 2024-05-08T00:54:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T00:54:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_2 2024-05-08T00:54:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:54:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:54:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:54:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:54:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=13 2024-05-08T00:54:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:54:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:55:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 52149248 bytes 2024-05-08T00:55:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 79173632 bytes 2024-05-08T00:55:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 105875456 bytes 2024-05-08T00:55:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 111980544 bytes 2024-05-08T00:55:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 164299776 bytes 2024-05-08T00:55:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_2 2024-05-08T00:55:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:55:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:55:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:55:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-05-08T00:55:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:55:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10ada52637b3 2024-05-08T00:55:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:55:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:55:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_2 present 10ada52637b3 2024-05-08T00:55:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:55:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:55:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T00:55:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10ada52637b3 2024-05-08T00:55:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:55:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_2 2024-05-08T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-05-08T00:55:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:55:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10ada52637b3 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10ada52637b3 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10ada52637b3 -o 10ada52637b3 = 10ada52637b3 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_2 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-05-08T00:55:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T00:55:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_2 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:55:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:55:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-05-08T00:55:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:55:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-05-08T00:55:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:55:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:55:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:55:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_2 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_2 2024-05-08T00:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-05-08T00:55:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:55:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-05-08T00:55:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:55:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:55:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:55:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:55:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_2 2024-05-08T00:55:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:55:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:55:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-05-08T00:55:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_2 2024-05-08T00:55:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:55:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:55:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:55:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:55:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_2 2024-05-08T00:55:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:55:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_2 2024-05-08T00:55:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-05-08T00:55:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-05-08T00:55:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T00:55:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:55:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:55:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_3 2024-05-08T00:55:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_3 512M 2024-05-08T00:55:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:55:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:55:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:55:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:55:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:55:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:55:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:55:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:55:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:55:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-05-08T00:55:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_3 2024-05-08T00:55:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:55:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:55:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:55:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 journal 2024-05-08T00:55:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T00:55:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-05-08T00:55:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_3': 2024-05-08T00:55:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:55:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1cabda9a1b6c 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1cabda9a1b6c 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:55:40 2024 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:55:40 2024 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:55:40 2024 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1cabda9a1b6c 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:55:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 2b281661-2b0b-4c9a-9322-5e491a1fd840 2024-05-08T00:55:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T00:55:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_3 2024-05-08T00:55:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:55:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:55:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:55:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:55:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:55:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:55:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 35461120 bytes 2024-05-08T00:55:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 72246272 bytes 2024-05-08T00:55:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 136143872 bytes 2024-05-08T00:55:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:55:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:55:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:55:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:55:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-05-08T00:55:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_3 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:55:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-05-08T00:55:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:55:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aa212cd674 2024-05-08T00:55:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:55:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:55:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_3 present 10aa212cd674 2024-05-08T00:55:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:55:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:55:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T00:55:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aa212cd674 2024-05-08T00:55:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:55:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:55:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_3 2024-05-08T00:55:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:55:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:55:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:55:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-05-08T00:55:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:55:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10aa212cd674 2024-05-08T00:55:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10aa212cd674 2024-05-08T00:55:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10aa212cd674 -o 10aa212cd674 = 10aa212cd674 2024-05-08T00:55:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:55:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T00:55:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:55:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_3 2024-05-08T00:55:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:55:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:55:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:55:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-05-08T00:56:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T00:56:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-05-08T00:56:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:56:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:56:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:56:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_3 2024-05-08T00:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-05-08T00:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:56:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-05-08T00:56:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:56:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:56:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:56:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-05-08T00:56:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_3 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_3 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-05-08T00:56:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:56:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-05-08T00:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_3 2024-05-08T00:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:56:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-05-08T00:56:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_3 2024-05-08T00:56:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:56:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:56:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:56:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:56:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_3 2024-05-08T00:56:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:56:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_3 2024-05-08T00:56:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-05-08T00:56:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-05-08T00:56:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T00:56:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:56:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:56:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_4 2024-05-08T00:56:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_4 512M 2024-05-08T00:56:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:56:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:56:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:56:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:56:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:56:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:56:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:56:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:56:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:56:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-05-08T00:56:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_4 2024-05-08T00:56:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:56:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 journal 2024-05-08T00:56:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T00:56:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_4': 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1cc3e8b1ed2b 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1cc3e8b1ed2b 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:56:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:56:22 2024 2024-05-08T00:56:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:56:22 2024 2024-05-08T00:56:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:56:22 2024 2024-05-08T00:56:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1cc3e8b1ed2b 2024-05-08T00:56:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:56:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:56:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 27057f02-6aaa-466e-8039-9e0aeb06c0f0 2024-05-08T00:56:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T00:56:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_4 2024-05-08T00:56:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:56:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:56:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:56:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=11 2024-05-08T00:56:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:56:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:56:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 51928064 bytes 2024-05-08T00:56:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 104320000 bytes 2024-05-08T00:56:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 149113856 bytes 2024-05-08T00:56:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 172613632 bytes 2024-05-08T00:56:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:56:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:56:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:56:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_4 2024-05-08T00:56:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:56:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:56:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:56:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-05-08T00:56:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:56:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aa6d425273 2024-05-08T00:56:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:56:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:56:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_4 present 10aa6d425273 2024-05-08T00:56:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:56:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:56:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T00:56:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aa6d425273 2024-05-08T00:56:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:56:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:56:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_4 2024-05-08T00:56:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:56:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:56:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:56:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-05-08T00:56:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:56:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10aa6d425273 2024-05-08T00:56:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10aa6d425273 2024-05-08T00:56:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10aa6d425273 -o 10aa6d425273 = 10aa6d425273 2024-05-08T00:56:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:56:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T00:56:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:56:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_4 2024-05-08T00:56:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:56:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:56:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:56:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-05-08T00:57:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_4 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:57:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:57:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:57:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-05-08T00:57:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:57:03.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-05-08T00:57:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:57:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:57:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:57:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-05-08T00:57:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:57:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:57:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:57:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:57:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:57:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:57:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_4 2024-05-08T00:57:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:57:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:57:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_4 2024-05-08T00:57:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-05-08T00:57:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:57:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-05-08T00:57:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:57:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:57:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:57:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:57:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_4 2024-05-08T00:57:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:57:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-05-08T00:57:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_4 2024-05-08T00:57:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:57:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:57:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:57:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_4 2024-05-08T00:57:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_4 2024-05-08T00:57:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-05-08T00:57:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-05-08T00:57:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_5 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_5 512M 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:57:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:57:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:57:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-05-08T00:57:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_5 2024-05-08T00:57:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:57:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 journal 2024-05-08T00:57:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T00:57:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-05-08T00:57:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_5': 2024-05-08T00:57:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:57:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:57:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T00:57:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1cdbeb9e19b4 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1cdbeb9e19b4 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:57:04 2024 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:57:04 2024 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:57:04 2024 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1cdbeb9e19b4 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:57:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 82cdf64d-8822-47c0-afb2-b22bc924e555 2024-05-08T00:57:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T00:57:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_5 2024-05-08T00:57:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:57:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=14 2024-05-08T00:57:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:57:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:57:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 61043712 bytes 2024-05-08T00:57:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 93521920 bytes 2024-05-08T00:57:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 127173632 bytes 2024-05-08T00:57:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 139511808 bytes 2024-05-08T00:57:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 177178624 bytes 2024-05-08T00:57:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:600 IOs, 184956928 bytes 2024-05-08T00:57:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:700 IOs, 221505536 bytes 2024-05-08T00:57:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_5 2024-05-08T00:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-05-08T00:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:57:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aad39f51ee 2024-05-08T00:57:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:57:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:57:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_5 present 10aad39f51ee 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aad39f51ee 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_5 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-05-08T00:57:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:57:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10aad39f51ee 2024-05-08T00:57:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10aad39f51ee 2024-05-08T00:57:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10aad39f51ee -o 10aad39f51ee = 10aad39f51ee 2024-05-08T00:57:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:57:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T00:57:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:57:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_5 2024-05-08T00:57:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-05-08T00:57:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T00:57:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-05-08T00:57:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:57:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:57:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:57:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:57:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:57:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_5 2024-05-08T00:57:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:57:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:57:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:57:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-05-08T00:57:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:57:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-05-08T00:57:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:57:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:57:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:57:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_5 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_5 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-05-08T00:57:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:57:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-05-08T00:57:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:57:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:57:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:57:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:57:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_5 2024-05-08T00:57:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-05-08T00:57:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_5 2024-05-08T00:57:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:57:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:57:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_5 2024-05-08T00:57:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_5 2024-05-08T00:57:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-05-08T00:57:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-05-08T00:57:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_6 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_6 512M 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:57:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:57:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:57:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:57:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-05-08T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_6 2024-05-08T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:57:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 journal 2024-05-08T00:57:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T00:57:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_6': 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1cf3ded2b99c 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1cf3ded2b99c 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:57:49 2024 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:57:49 2024 2024-05-08T00:57:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:57:49 2024 2024-05-08T00:57:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1cf3ded2b99c 2024-05-08T00:57:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:57:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:57:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 6075363b-caae-404b-ac21-60d8d534ed44 2024-05-08T00:57:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T00:57:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_6 2024-05-08T00:57:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:57:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:57:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:57:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:57:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=13 2024-05-08T00:57:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:57:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:57:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 32066560 bytes 2024-05-08T00:57:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 81841152 bytes 2024-05-08T00:57:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 115720192 bytes 2024-05-08T00:57:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 116323328 bytes 2024-05-08T00:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:500 IOs, 161940480 bytes 2024-05-08T00:58:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:58:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:58:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_6 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:58:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:58:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-05-08T00:58:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:58:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aa282e4c4e 2024-05-08T00:58:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:58:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:58:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_6 present 10aa282e4c4e 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aa282e4c4e 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_6 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:58:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-05-08T00:58:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10aa282e4c4e 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10aa282e4c4e 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10aa282e4c4e -o 10aa282e4c4e = 10aa282e4c4e 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_6 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:58:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-05-08T00:58:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T00:58:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_6 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:58:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-05-08T00:58:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:58:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-05-08T00:58:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:58:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:58:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:58:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-05-08T00:58:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:58:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_6 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_6 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-05-08T00:58:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:58:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-05-08T00:58:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:58:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:58:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:58:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:58:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_6 2024-05-08T00:58:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:58:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:58:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-05-08T00:58:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_6 2024-05-08T00:58:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:58:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:58:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:58:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:58:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_6 2024-05-08T00:58:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:58:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_6 2024-05-08T00:58:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-05-08T00:58:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-05-08T00:58:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_7 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_7 512M 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:58:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:58:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:58:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:58:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-05-08T00:58:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_7 2024-05-08T00:58:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:58:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:58:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:58:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 journal 2024-05-08T00:58:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T00:58:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-05-08T00:58:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_7': 2024-05-08T00:58:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:58:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:58:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T00:58:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1d0baf4a4d27 2024-05-08T00:58:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1d0baf4a4d27 2024-05-08T00:58:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:58:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:58:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:58:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:58:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:58:31 2024 2024-05-08T00:58:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:58:31 2024 2024-05-08T00:58:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:58:31 2024 2024-05-08T00:58:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1d0baf4a4d27 2024-05-08T00:58:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:58:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:58:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: e69b341b-a0dd-4893-9862-13d304925e3e 2024-05-08T00:58:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T00:58:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_7 2024-05-08T00:58:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:58:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:58:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:58:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=5 2024-05-08T00:58:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:58:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:58:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 38966272 bytes 2024-05-08T00:58:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 53827584 bytes 2024-05-08T00:58:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_7 2024-05-08T00:58:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:58:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:58:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:58:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-05-08T00:58:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:58:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aaf5385ae9 2024-05-08T00:58:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:58:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:58:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_7 present 10aaf5385ae9 2024-05-08T00:58:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:58:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:58:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T00:58:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aaf5385ae9 2024-05-08T00:58:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:58:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:58:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_7 2024-05-08T00:58:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:58:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:58:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:58:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-05-08T00:58:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10aaf5385ae9 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10aaf5385ae9 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10aaf5385ae9 -o 10aaf5385ae9 = 10aaf5385ae9 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_7 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:58:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:58:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-05-08T00:59:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_7 2024-05-08T00:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:59:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:59:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:59:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-05-08T00:59:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:59:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aaf5385ae9 2024-05-08T00:59:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:59:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_7 deleted 10aaf5385ae9 2024-05-08T00:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aaf5385ae9 2024-05-08T00:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:59:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_7 2024-05-08T00:59:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:59:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:59:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_7 2024-05-08T00:59:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-05-08T00:59:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:59:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-05-08T00:59:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:59:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:59:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:59:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:59:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_7 2024-05-08T00:59:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:59:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-05-08T00:59:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_7 2024-05-08T00:59:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:59:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:59:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:59:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_7 2024-05-08T00:59:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_7 2024-05-08T00:59:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-05-08T00:59:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-05-08T00:59:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T00:59:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:59:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_8 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_8 512M 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:59:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:59:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-05-08T00:59:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_8 2024-05-08T00:59:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:59:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 journal 2024-05-08T00:59:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T00:59:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-05-08T00:59:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_8': 2024-05-08T00:59:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:59:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:59:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T00:59:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1d22d7eb9d9 2024-05-08T00:59:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1d22d7eb9d9 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:59:02 2024 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:59:02 2024 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:59:02 2024 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1d22d7eb9d9 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: f99c09db-3181-4c37-ac16-7ddf03193d81 2024-05-08T00:59:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T00:59:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_8 2024-05-08T00:59:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:59:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=13 2024-05-08T00:59:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:59:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:59:04.226 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 30054400 bytes 2024-05-08T00:59:07.340 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 121356288 bytes 2024-05-08T00:59:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 156684288 bytes 2024-05-08T00:59:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:400 IOs, 157302784 bytes 2024-05-08T00:59:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_8 2024-05-08T00:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-05-08T00:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:59:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10ad178a29ce 2024-05-08T00:59:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:59:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_8 present 10ad178a29ce 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10ad178a29ce 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_8 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:59:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-05-08T00:59:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:59:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10ad178a29ce 2024-05-08T00:59:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10ad178a29ce 2024-05-08T00:59:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10ad178a29ce -o 10ad178a29ce = 10ad178a29ce 2024-05-08T00:59:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:59:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T00:59:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:59:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_8 2024-05-08T00:59:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-05-08T00:59:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T00:59:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_8 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-05-08T00:59:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:59:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-05-08T00:59:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T00:59:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:59:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:59:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-05-08T00:59:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:59:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T00:59:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:59:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:59:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:59:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_8 2024-05-08T00:59:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:59:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:59:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_8 2024-05-08T00:59:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-05-08T00:59:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:59:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-05-08T00:59:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T00:59:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:59:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T00:59:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:59:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_8 2024-05-08T00:59:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-05-08T00:59:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_8 2024-05-08T00:59:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T00:59:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T00:59:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_8 2024-05-08T00:59:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_8 2024-05-08T00:59:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-05-08T00:59:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-05-08T00:59:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_9 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_9 512M 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T00:59:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T00:59:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T00:59:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T00:59:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-05-08T00:59:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_9 2024-05-08T00:59:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:59:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T00:59:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 journal 2024-05-08T00:59:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T00:59:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-05-08T00:59:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_9': 2024-05-08T00:59:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T00:59:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T00:59:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1d3a77004832 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1d3a77004832 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 00:59:46 2024 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 00:59:46 2024 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 00:59:46 2024 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1d3a77004832 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 7d53a142-85d4-4f9a-9d19-d53ae34fe4e2 2024-05-08T00:59:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T00:59:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_9 2024-05-08T00:59:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:59:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T00:59:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=9 2024-05-08T00:59:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T00:59:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-05-08T00:59:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 25074688 bytes 2024-05-08T00:59:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:200 IOs, 43802624 bytes 2024-05-08T00:59:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:300 IOs, 88987648 bytes 2024-05-08T00:59:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T00:59:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_9 2024-05-08T00:59:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:59:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:59:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T00:59:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-05-08T00:59:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:59:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aa964738cb 2024-05-08T00:59:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T00:59:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T00:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_9 present 10aa964738cb 2024-05-08T00:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T00:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T00:59:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aa964738cb 2024-05-08T00:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T00:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T00:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_9 2024-05-08T00:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T00:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T00:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T00:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-05-08T00:59:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T00:59:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10aa964738cb 2024-05-08T00:59:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10aa964738cb 2024-05-08T00:59:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10aa964738cb -o 10aa964738cb = 10aa964738cb 2024-05-08T00:59:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T00:59:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T00:59:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T00:59:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_9 2024-05-08T00:59:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T00:59:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T00:59:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T00:59:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-05-08T01:00:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T01:00:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-05-08T01:00:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:00:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_9 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-05-08T01:00:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T01:00:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-05-08T01:00:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T01:00:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T01:00:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T01:00:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-05-08T01:00:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:00:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T01:00:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T01:00:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T01:00:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T01:00:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T01:00:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_9 2024-05-08T01:00:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:00:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:00:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_9 2024-05-08T01:00:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-05-08T01:00:25.935 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T01:00:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-05-08T01:00:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T01:00:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T01:00:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T01:00:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:00:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_9 2024-05-08T01:00:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:00:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T01:00:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-05-08T01:00:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_9 2024-05-08T01:00:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:00:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T01:00:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T01:00:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T01:00:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_9 2024-05-08T01:00:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:00:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_9 2024-05-08T01:00:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-05-08T01:00:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-05-08T01:00:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T01:00:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for i in `seq 1 10` 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image=image_10 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ create_image cluster2 mirror image_10 512M 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local size=128 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 512M ']' 2024-05-08T01:00:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ size=512M 2024-05-08T01:00:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ shift 2024-05-08T01:00:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-05-08T01:00:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ enable_mirror cluster2 mirror image_10 2024-05-08T01:00:26.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:00:26.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:26.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:00:26.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local mode=journal 2024-05-08T01:00:26.325 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 journal 2024-05-08T01:00:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring enabled 2024-05-08T01:00:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-05-08T01:00:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:rbd image 'image_10': 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: size 512 MiB in 128 objects 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: order 22 (4 MiB objects) 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: snapshot_count: 0 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: id: 1d52e43810f0 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: block_name_prefix: rbd_data.1d52e43810f0 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: format: 2 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: features: layering, exclusive-lock, journaling 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: op_features: 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: flags: 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: create_timestamp: Wed May 8 01:00:26 2024 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: access_timestamp: Wed May 8 01:00:26 2024 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: modify_timestamp: Wed May 8 01:00:26 2024 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: journal: 1d52e43810f0 2024-05-08T01:00:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring state: enabled 2024-05-08T01:00:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring mode: journal 2024-05-08T01:00:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring global id: 3c605936-fdc8-4481-b312-3e8a9f491fef 2024-05-08T01:00:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout: mirroring primary: true 2024-05-08T01:00:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stress_write_image cluster2 mirror image_10 2024-05-08T01:00:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:00:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:00:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-08T01:00:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local duration=7 2024-05-08T01:00:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T01:00:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-05-08T01:00:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:100 IOs, 33137664 bytes 2024-05-08T01:00:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=124 2024-05-08T01:00:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T01:00:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 124 -eq 124 ']' 2024-05-08T01:00:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:00:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-05-08T01:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=present 2024-05-08T01:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T01:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T01:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T01:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_10 2024-05-08T01:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T01:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-05-08T01:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T01:00:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id=10aa57d4eafe 2024-05-08T01:00:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T01:00:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T01:00:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_10 present 10aa57d4eafe 2024-05-08T01:00:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:00:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=present 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id=10aa57d4eafe 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_10 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-05-08T01:00:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id=10aa57d4eafe 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n 10aa57d4eafe 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -z 10aa57d4eafe -o 10aa57d4eafe = 10aa57d4eafe 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_state=present 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test present = present 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ disable_mirror cluster2 mirror image_10 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:00:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-05-08T01:01:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:Mirroring disabled 2024-05-08T01:01:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local state=deleted 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local s 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_10 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-05-08T01:01:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T01:01:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-05-08T01:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ image_id= 2024-05-08T01:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-08T01:01:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0.1 2024-05-08T01:01:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-05-08T01:01:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:01:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:01:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local test_state=deleted 2024-05-08T01:01:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image_id= 2024-05-08T01:01:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_state=deleted 2024-05-08T01:01:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local current_image_id 2024-05-08T01:01:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ get_image_id cluster1 mirror image_10 2024-05-08T01:01:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local image=image_10 2024-05-08T01:01:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-05-08T01:01:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-08T01:01:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-05-08T01:01:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ current_image_id= 2024-05-08T01:01:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T01:01:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test deleted = deleted 2024-05-08T01:01:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:01:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ purge_snapshots cluster2 mirror image_10 2024-05-08T01:01:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:01:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:01:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-05-08T01:01:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image_retry cluster2 mirror image_10 2024-05-08T01:01:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:01:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:01:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-08T01:01:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T01:01:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ remove_image cluster2 mirror image_10 2024-05-08T01:01:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:01:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local image=image_10 2024-05-08T01:01:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-05-08T01:01:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-05-08T01:01:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-08T01:01:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:01:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-05-08T01:01:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ tee -a /tmp/tmp.W1VfRby4SU/rbd-mirror.test.log 2024-05-08T01:01:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ date '+%F %T' 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo 2024-05-08 01:01:04 TEST: check if removed 'images'\''' OMAP are removed 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:2024-05-08 01:01:04 TEST: check if removed images' OMAP are removed 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local obj_name=rbd_mirroring 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local filter=status_global 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-05-08T01:01:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T01:01:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirroring 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local filter=status_global 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirroring 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-05-08T01:01:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -c status_global 2024-05-08T01:01:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 = 0 2024-05-08T01:01:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T01:01:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 0 ']' 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local obj_name=rbd_mirroring 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local filter=image_ 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T01:01:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T01:01:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-05-08T01:01:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirroring 2024-05-08T01:01:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local filter=image_ 2024-05-08T01:01:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-05-08T01:01:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirroring 2024-05-08T01:01:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-05-08T01:01:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -c image_ 2024-05-08T01:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 = 0 2024-05-08T01:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T01:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T01:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 0 ']' 2024-05-08T01:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-05-08T01:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local obj_name=rbd_mirror_leader 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local filter=image_map 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirror_leader 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local filter=image_map 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirror_leader 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-05-08T01:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -c image_map 2024-05-08T01:01:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 = 0 2024-05-08T01:01:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T01:01:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 0 ']' 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local obj_name=rbd_mirroring 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local filter=status_global 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-05-08T01:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirroring 2024-05-08T01:01:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local filter=status_global 2024-05-08T01:01:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-05-08T01:01:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirroring 2024-05-08T01:01:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-05-08T01:01:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -c status_global 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 = 0 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 0 ']' 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local obj_name=rbd_mirroring 2024-05-08T01:01:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local filter=image_ 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirroring 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local filter=image_ 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirroring 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-05-08T01:01:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -c image_ 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 = 0 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 0 ']' 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pool=mirror 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local obj_name=rbd_mirror_leader 2024-05-08T01:01:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local filter=image_map 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 0 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirror_leader 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local filter=image_map 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local pool=mirror 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local obj_name=rbd_mirror_leader 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-05-08T01:01:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ grep -c image_map 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test 0 = 0 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ error_code=0 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set -e 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 0 ']' 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ return 0 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ cleanup 0 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local error_code=0 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ set +e 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -ne 0 ']' 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -z '' ']' 2024-05-08T01:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster instance 2024-05-08T01:01:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T01:01:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-05-08T01:01:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:pool 'mirror' does not exist 2024-05-08T01:01:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T01:01:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-05-08T01:01:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:pool 'mirror' does not exist 2024-05-08T01:01:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T01:01:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-05-08T01:01:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:pool 'mirror_parent' does not exist 2024-05-08T01:01:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ CEPH_ARGS= 2024-05-08T01:01:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-05-08T01:01:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:pool 'mirror_parent' does not exist 2024-05-08T01:01:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-05-08T01:01:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stop_mirrors cluster1 2024-05-08T01:01:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1 2024-05-08T01:01:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local sig= 2024-05-08T01:01:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T01:01:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T01:01:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stop_mirror cluster1:0 '' 2024-05-08T01:01:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:0 2024-05-08T01:01:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local sig= 2024-05-08T01:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T01:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pid 2024-05-08T01:01:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_pid_file cluster1:0 2024-05-08T01:01:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1:0 2024-05-08T01:01:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:0 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=cluster 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval cluster=cluster1 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ cluster=cluster1 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T01:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-05-08T01:01:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-05-08T01:01:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-05-08T01:01:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ pid=36911 2024-05-08T01:01:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 36911 ']' 2024-05-08T01:01:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ kill 36911 2024-05-08T01:01:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 16 32 2024-05-08T01:01:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T01:01:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ps auxww 2024-05-08T01:01:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ awk -v pid=36911 '$2 == pid {print; exit 1}' 2024-05-08T01:01:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T01:01:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ps auxww 2024-05-08T01:01:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ awk -v pid=36911 '$2 == pid {print; exit 1}' 2024-05-08T01:01:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-05-08T01:01:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T01:01:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T01:01:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T01:01:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T01:01:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:0 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T01:01:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T01:01:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T01:01:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T01:01:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T01:01:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T01:01:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T01:01:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-08T01:01:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T01:01:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-08T01:01:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_pid_file cluster1:0 2024-05-08T01:01:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:0 2024-05-08T01:01:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:0 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T01:01:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-05-08T01:01:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-05-08T01:01:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-05-08T01:01:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T01:01:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stop_mirror cluster1:1 '' 2024-05-08T01:01:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster1:1 2024-05-08T01:01:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local sig= 2024-05-08T01:01:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pid 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_pid_file cluster1:1 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster1:1 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster1:1 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=cluster 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval cluster=cluster1 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ cluster=cluster1 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T01:01:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T01:01:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-05-08T01:01:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-05-08T01:01:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-05-08T01:01:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ pid=36939 2024-05-08T01:01:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 36939 ']' 2024-05-08T01:01:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ kill 36939 2024-05-08T01:01:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 16 32 2024-05-08T01:01:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T01:01:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ps auxww 2024-05-08T01:01:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ awk -v pid=36939 '$2 == pid {print; exit 1}' 2024-05-08T01:01:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T01:01:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ps auxww 2024-05-08T01:01:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ awk -v pid=36939 '$2 == pid {print; exit 1}' 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T01:01:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T01:01:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T01:01:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T01:01:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T01:01:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T01:01:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T01:01:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T01:01:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-05-08T01:01:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster1:1 2024-05-08T01:01:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-08T01:01:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T01:01:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T01:01:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T01:01:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T01:01:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T01:01:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster1 2024-05-08T01:01:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster1 2024-05-08T01:01:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T01:01:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T01:01:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-08T01:01:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T01:01:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-08T01:01:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_pid_file cluster1:1 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1:1 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster1:1 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster1 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster1 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster1 2024-05-08T01:01:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T01:01:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T01:01:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-05-08T01:01:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-05-08T01:01:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-05-08T01:01:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-05-08T01:01:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stop_mirrors cluster2 2024-05-08T01:01:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2 2024-05-08T01:01:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local sig= 2024-05-08T01:01:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ seq 0 1 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stop_mirror cluster2:0 '' 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2:0 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local sig= 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pid 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_pid_file cluster2:0 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster2:0 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-05-08T01:01:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster2:0 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=cluster 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster2 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 0 = cluster2:0 ']' 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval cluster=cluster2 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ cluster=cluster2 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=0 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=0 2024-05-08T01:01:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-05-08T01:01:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-05-08T01:01:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-05-08T01:01:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ pid=37007 2024-05-08T01:01:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 37007 ']' 2024-05-08T01:01:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ kill 37007 2024-05-08T01:01:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 16 32 2024-05-08T01:01:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T01:01:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ps auxww 2024-05-08T01:01:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ awk -v pid=37007 '$2 == pid {print; exit 1}' 2024-05-08T01:01:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T01:01:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ps auxww 2024-05-08T01:01:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ awk -v pid=37007 '$2 == pid {print; exit 1}' 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster2:0 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster2:0 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster2 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster2:0 ']' 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster2 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster2 2024-05-08T01:01:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T01:01:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T01:01:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-05-08T01:01:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-05-08T01:01:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster2:0 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster2:0 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster2 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T01:01:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster2:0 ']' 2024-05-08T01:01:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster2 2024-05-08T01:01:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster2 2024-05-08T01:01:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T01:01:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T01:01:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-05-08T01:01:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-05-08T01:01:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-05-08T01:01:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_pid_file cluster2:0 2024-05-08T01:01:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2:0 2024-05-08T01:01:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster2:0 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster2 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=0 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 0 = cluster2:0 ']' 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster2 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster2 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=0 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=0 2024-05-08T01:01:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-05-08T01:01:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-05-08T01:01:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-05-08T01:01:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-08T01:01:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ stop_mirror cluster2:1 '' 2024-05-08T01:01:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local cluster=cluster2:1 2024-05-08T01:01:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local sig= 2024-05-08T01:01:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test -n '' 2024-05-08T01:01:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ local pid 2024-05-08T01:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ daemon_pid_file cluster2:1 2024-05-08T01:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster=cluster2:1 2024-05-08T01:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance 2024-05-08T01:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2024-05-08T01:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local val=cluster2:1 2024-05-08T01:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local cluster_var_name=cluster 2024-05-08T01:01:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local instance_var_name=instance 2024-05-08T01:01:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster2 2024-05-08T01:01:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T01:01:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ '[' 1 = cluster2:1 ']' 2024-05-08T01:01:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval cluster=cluster2 2024-05-08T01:01:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ cluster=cluster2 2024-05-08T01:01:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ eval instance=1 2024-05-08T01:01:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ instance=1 2024-05-08T01:01:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-05-08T01:01:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-05-08T01:01:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-05-08T01:01:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ pid=37068 2024-05-08T01:01:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -n 37068 ']' 2024-05-08T01:01:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ kill 37068 2024-05-08T01:01:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ for s in 1 2 4 8 16 32 2024-05-08T01:01:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ sleep 1 2024-05-08T01:01:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ps auxww 2024-05-08T01:01:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ awk -v pid=37068 '$2 == pid {print; exit 1}' 2024-05-08T01:01:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ break 2024-05-08T01:01:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ ps auxww 2024-05-08T01:01:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ awk -v pid=37068 '$2 == pid {print; exit 1}' 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster2:1 cluster1 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster2:1 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster1 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster2:1 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster2 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster2:1 ']' 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster2 2024-05-08T01:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster2 2024-05-08T01:01:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T01:01:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T01:01:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-05-08T01:01:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-05-08T01:01:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-05-08T01:01:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-05-08T01:01:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local local_cluster=cluster2:1 2024-05-08T01:01:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2 2024-05-08T01:01:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-05-08T01:01:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster2:1 2024-05-08T01:01:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=local_cluster 2024-05-08T01:01:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster2 2024-05-08T01:01:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T01:01:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster2:1 ']' 2024-05-08T01:01:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval local_cluster=cluster2 2024-05-08T01:01:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ local_cluster=cluster2 2024-05-08T01:01:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T01:01:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T01:01:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-05-08T01:01:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-05-08T01:01:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-05-08T01:01:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ daemon_pid_file cluster2:1 2024-05-08T01:01:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster=cluster2:1 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local val=cluster2:1 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local cluster_var_name=cluster 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ local instance_var_name=instance 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ cluster=cluster2 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ instance=1 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ '[' 1 = cluster2:1 ']' 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval cluster=cluster2 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ cluster=cluster2 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ eval instance=1 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ instance=1 2024-05-08T01:01:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-05-08T01:01:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-05-08T01:01:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-05-08T01:01:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' -z 1 ']' 2024-05-08T01:01:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ test '' = /tmp/tmp.W1VfRby4SU 2024-05-08T01:01:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ rm -Rf /tmp/tmp.W1VfRby4SU 2024-05-08T01:01:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ '[' 0 -eq 0 ']' 2024-05-08T01:01:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ echo OK 2024-05-08T01:01:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stdout:OK 2024-05-08T01:01:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi116.stderr:+ exit 0 2024-05-08T01:01:13.276 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-08T01:01:13.276 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-05-08T01:01:13.450 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-05-08T01:01:13.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-05-08T01:01:13.703 DEBUG:teuthology.parallel:result is None 2024-05-08T01:01:13.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-05-08T01:01:13.728 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-05-08T01:01:13.728 DEBUG:teuthology.orchestra.run.smithi116:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-05-08T01:01:13.783 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-05-08T01:01:13.783 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-08T01:01:13.795 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-05-08T01:01:13.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-08T01:01:14.108 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T01:01:14.108 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-05-08T01:01:14.120 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":1353,"stamp":"2024-05-08T01:01:12.417240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3051996,"kb_used_data":14712,"kb_used_omap":381,"kb_used_meta":3036866,"kb_avail":280063524,"statfs":{"total":289910292480,"available":286785048576,"internally_reserved":0,"allocated":15065088,"data_stored":8181863,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":391062,"internal_metadata":3109750890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-389171254,"num_objects":-133,"num_object_clones":0,"num_object_copies":-266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-133,"num_whiteouts":0,"num_read":-839226,"num_read_kb":-43245652,"num_write":-1240645,"num_write_kb":-23898360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-82944,"num_keys_recovered":-2,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002034"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.345582+0000","last_change":"2024-05-08T00:30:09.022863+0000","last_active":"2024-05-08T01:01:09.345582+0000","last_peered":"2024-05-08T01:01:09.345582+0000","last_clean":"2024-05-08T01:01:09.345582+0000","last_became_active":"2024-05-08T00:30:09.022284+0000","last_became_peered":"2024-05-08T00:30:09.022284+0000","last_unstale":"2024-05-08T01:01:09.345582+0000","last_undegraded":"2024-05-08T01:01:09.345582+0000","last_fullsized":"2024-05-08T01:01:09.345582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:22:13.046711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.346202+0000","last_change":"2024-05-08T00:30:09.022565+0000","last_active":"2024-05-08T01:01:09.346202+0000","last_peered":"2024-05-08T01:01:09.346202+0000","last_clean":"2024-05-08T01:01:09.346202+0000","last_became_active":"2024-05-08T00:30:09.022263+0000","last_became_peered":"2024-05-08T00:30:09.022263+0000","last_unstale":"2024-05-08T01:01:09.346202+0000","last_undegraded":"2024-05-08T01:01:09.346202+0000","last_fullsized":"2024-05-08T01:01:09.346202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:16:17.936696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347366+0000","last_change":"2024-05-08T00:30:09.022872+0000","last_active":"2024-05-08T01:01:09.347366+0000","last_peered":"2024-05-08T01:01:09.347366+0000","last_clean":"2024-05-08T01:01:09.347366+0000","last_became_active":"2024-05-08T00:30:09.022298+0000","last_became_peered":"2024-05-08T00:30:09.022298+0000","last_unstale":"2024-05-08T01:01:09.347366+0000","last_undegraded":"2024-05-08T01:01:09.347366+0000","last_fullsized":"2024-05-08T01:01:09.347366+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:03:16.834391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347181+0000","last_change":"2024-05-08T00:30:09.022579+0000","last_active":"2024-05-08T01:01:09.347181+0000","last_peered":"2024-05-08T01:01:09.347181+0000","last_clean":"2024-05-08T01:01:09.347181+0000","last_became_active":"2024-05-08T00:30:09.022205+0000","last_became_peered":"2024-05-08T00:30:09.022205+0000","last_unstale":"2024-05-08T01:01:09.347181+0000","last_undegraded":"2024-05-08T01:01:09.347181+0000","last_fullsized":"2024-05-08T01:01:09.347181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:14:39.237433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":835,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:08.676118+0000","last_change":"2024-05-08T00:30:09.023296+0000","last_active":"2024-05-08T01:01:08.676118+0000","last_peered":"2024-05-08T01:01:08.676118+0000","last_clean":"2024-05-08T01:01:08.676118+0000","last_became_active":"2024-05-08T00:30:09.023003+0000","last_became_peered":"2024-05-08T00:30:09.023003+0000","last_unstale":"2024-05-08T01:01:08.676118+0000","last_undegraded":"2024-05-08T01:01:08.676118+0000","last_fullsized":"2024-05-08T01:01:08.676118+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:21:27.320591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":835,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.361667+0000","last_change":"2024-05-08T00:30:09.023325+0000","last_active":"2024-05-08T01:01:09.361667+0000","last_peered":"2024-05-08T01:01:09.361667+0000","last_clean":"2024-05-08T01:01:09.361667+0000","last_became_active":"2024-05-08T00:30:09.022878+0000","last_became_peered":"2024-05-08T00:30:09.022878+0000","last_unstale":"2024-05-08T01:01:09.361667+0000","last_undegraded":"2024-05-08T01:01:09.361667+0000","last_fullsized":"2024-05-08T01:01:09.361667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:18:31.386016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.361976+0000","last_change":"2024-05-08T00:30:09.024010+0000","last_active":"2024-05-08T01:01:09.361976+0000","last_peered":"2024-05-08T01:01:09.361976+0000","last_clean":"2024-05-08T01:01:09.361976+0000","last_became_active":"2024-05-08T00:30:09.023682+0000","last_became_peered":"2024-05-08T00:30:09.023682+0000","last_unstale":"2024-05-08T01:01:09.361976+0000","last_undegraded":"2024-05-08T01:01:09.361976+0000","last_fullsized":"2024-05-08T01:01:09.361976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:07:10.443006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347317+0000","last_change":"2024-05-08T00:30:09.023043+0000","last_active":"2024-05-08T01:01:09.347317+0000","last_peered":"2024-05-08T01:01:09.347317+0000","last_clean":"2024-05-08T01:01:09.347317+0000","last_became_active":"2024-05-08T00:30:09.022369+0000","last_became_peered":"2024-05-08T00:30:09.022369+0000","last_unstale":"2024-05-08T01:01:09.347317+0000","last_undegraded":"2024-05-08T01:01:09.347317+0000","last_fullsized":"2024-05-08T01:01:09.347317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:39:44.848225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":970,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347290+0000","last_change":"2024-05-08T00:30:03.285026+0000","last_active":"2024-05-08T01:01:09.347290+0000","last_peered":"2024-05-08T01:01:09.347290+0000","last_clean":"2024-05-08T01:01:09.347290+0000","last_became_active":"2024-05-08T00:30:03.284046+0000","last_became_peered":"2024-05-08T00:30:03.284046+0000","last_unstale":"2024-05-08T01:01:09.347290+0000","last_undegraded":"2024-05-08T01:01:09.347290+0000","last_fullsized":"2024-05-08T01:01:09.347290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_clean_scrub_stamp":"2024-05-08T00:30:01.996661+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:24:42.092364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738745,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721356,"kb_used_data":4864,"kb_used_omap":132,"kb_used_meta":716347,"kb_avail":93650484,"statfs":{"total":96636764160,"available":95898095616,"internally_reserved":0,"allocated":4980736,"data_stored":2333709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136015,"internal_metadata":733539505},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738746,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1032392,"kb_used_data":4924,"kb_used_omap":111,"kb_used_meta":1027344,"kb_avail":93339448,"statfs":{"total":96636764160,"available":95579594752,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":113964,"internal_metadata":1052000980},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738746,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1298248,"kb_used_data":4924,"kb_used_omap":137,"kb_used_meta":1293174,"kb_avail":93073592,"statfs":{"total":96636764160,"available":95307358208,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":141083,"internal_metadata":1324210405},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:01:14.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-08T01:01:14.403 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T01:01:14.404 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-05-08T01:01:14.416 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":1354,"stamp":"2024-05-08T01:01:14.417715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3051996,"kb_used_data":14712,"kb_used_omap":381,"kb_used_meta":3036866,"kb_avail":280063524,"statfs":{"total":289910292480,"available":286785048576,"internally_reserved":0,"allocated":15065088,"data_stored":8181863,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":391062,"internal_metadata":3109750890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-26,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-839418,"num_read_kb":-43245824,"num_write":-1240769,"num_write_kb":-23898365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-82944,"num_keys_recovered":-2,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002042"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.345582+0000","last_change":"2024-05-08T00:30:09.022863+0000","last_active":"2024-05-08T01:01:09.345582+0000","last_peered":"2024-05-08T01:01:09.345582+0000","last_clean":"2024-05-08T01:01:09.345582+0000","last_became_active":"2024-05-08T00:30:09.022284+0000","last_became_peered":"2024-05-08T00:30:09.022284+0000","last_unstale":"2024-05-08T01:01:09.345582+0000","last_undegraded":"2024-05-08T01:01:09.345582+0000","last_fullsized":"2024-05-08T01:01:09.345582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:22:13.046711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.346202+0000","last_change":"2024-05-08T00:30:09.022565+0000","last_active":"2024-05-08T01:01:09.346202+0000","last_peered":"2024-05-08T01:01:09.346202+0000","last_clean":"2024-05-08T01:01:09.346202+0000","last_became_active":"2024-05-08T00:30:09.022263+0000","last_became_peered":"2024-05-08T00:30:09.022263+0000","last_unstale":"2024-05-08T01:01:09.346202+0000","last_undegraded":"2024-05-08T01:01:09.346202+0000","last_fullsized":"2024-05-08T01:01:09.346202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:16:17.936696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347366+0000","last_change":"2024-05-08T00:30:09.022872+0000","last_active":"2024-05-08T01:01:09.347366+0000","last_peered":"2024-05-08T01:01:09.347366+0000","last_clean":"2024-05-08T01:01:09.347366+0000","last_became_active":"2024-05-08T00:30:09.022298+0000","last_became_peered":"2024-05-08T00:30:09.022298+0000","last_unstale":"2024-05-08T01:01:09.347366+0000","last_undegraded":"2024-05-08T01:01:09.347366+0000","last_fullsized":"2024-05-08T01:01:09.347366+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:03:16.834391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347181+0000","last_change":"2024-05-08T00:30:09.022579+0000","last_active":"2024-05-08T01:01:09.347181+0000","last_peered":"2024-05-08T01:01:09.347181+0000","last_clean":"2024-05-08T01:01:09.347181+0000","last_became_active":"2024-05-08T00:30:09.022205+0000","last_became_peered":"2024-05-08T00:30:09.022205+0000","last_unstale":"2024-05-08T01:01:09.347181+0000","last_undegraded":"2024-05-08T01:01:09.347181+0000","last_fullsized":"2024-05-08T01:01:09.347181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:14:39.237433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":835,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:08.676118+0000","last_change":"2024-05-08T00:30:09.023296+0000","last_active":"2024-05-08T01:01:08.676118+0000","last_peered":"2024-05-08T01:01:08.676118+0000","last_clean":"2024-05-08T01:01:08.676118+0000","last_became_active":"2024-05-08T00:30:09.023003+0000","last_became_peered":"2024-05-08T00:30:09.023003+0000","last_unstale":"2024-05-08T01:01:08.676118+0000","last_undegraded":"2024-05-08T01:01:08.676118+0000","last_fullsized":"2024-05-08T01:01:08.676118+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:21:27.320591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":835,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.361667+0000","last_change":"2024-05-08T00:30:09.023325+0000","last_active":"2024-05-08T01:01:09.361667+0000","last_peered":"2024-05-08T01:01:09.361667+0000","last_clean":"2024-05-08T01:01:09.361667+0000","last_became_active":"2024-05-08T00:30:09.022878+0000","last_became_peered":"2024-05-08T00:30:09.022878+0000","last_unstale":"2024-05-08T01:01:09.361667+0000","last_undegraded":"2024-05-08T01:01:09.361667+0000","last_fullsized":"2024-05-08T01:01:09.361667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:18:31.386016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.361976+0000","last_change":"2024-05-08T00:30:09.024010+0000","last_active":"2024-05-08T01:01:09.361976+0000","last_peered":"2024-05-08T01:01:09.361976+0000","last_clean":"2024-05-08T01:01:09.361976+0000","last_became_active":"2024-05-08T00:30:09.023682+0000","last_became_peered":"2024-05-08T00:30:09.023682+0000","last_unstale":"2024-05-08T01:01:09.361976+0000","last_undegraded":"2024-05-08T01:01:09.361976+0000","last_fullsized":"2024-05-08T01:01:09.361976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:07:10.443006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347317+0000","last_change":"2024-05-08T00:30:09.023043+0000","last_active":"2024-05-08T01:01:09.347317+0000","last_peered":"2024-05-08T01:01:09.347317+0000","last_clean":"2024-05-08T01:01:09.347317+0000","last_became_active":"2024-05-08T00:30:09.022369+0000","last_became_peered":"2024-05-08T00:30:09.022369+0000","last_unstale":"2024-05-08T01:01:09.347317+0000","last_undegraded":"2024-05-08T01:01:09.347317+0000","last_fullsized":"2024-05-08T01:01:09.347317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:39:44.848225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":970,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347290+0000","last_change":"2024-05-08T00:30:03.285026+0000","last_active":"2024-05-08T01:01:09.347290+0000","last_peered":"2024-05-08T01:01:09.347290+0000","last_clean":"2024-05-08T01:01:09.347290+0000","last_became_active":"2024-05-08T00:30:03.284046+0000","last_became_peered":"2024-05-08T00:30:03.284046+0000","last_unstale":"2024-05-08T01:01:09.347290+0000","last_undegraded":"2024-05-08T01:01:09.347290+0000","last_fullsized":"2024-05-08T01:01:09.347290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_clean_scrub_stamp":"2024-05-08T00:30:01.996661+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:24:42.092364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738745,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721356,"kb_used_data":4864,"kb_used_omap":132,"kb_used_meta":716347,"kb_avail":93650484,"statfs":{"total":96636764160,"available":95898095616,"internally_reserved":0,"allocated":4980736,"data_stored":2333709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136015,"internal_metadata":733539505},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738746,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1032392,"kb_used_data":4924,"kb_used_omap":111,"kb_used_meta":1027344,"kb_avail":93339448,"statfs":{"total":96636764160,"available":95579594752,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":113964,"internal_metadata":1052000980},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738746,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1298248,"kb_used_data":4924,"kb_used_omap":137,"kb_used_meta":1293174,"kb_avail":93073592,"statfs":{"total":96636764160,"available":95307358208,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":141083,"internal_metadata":1324210405},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:01:14.417 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-05-08T01:01:14.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-08T01:01:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T01:01:14.698 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-05-08T01:01:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":1354,"stamp":"2024-05-08T01:01:14.417715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3051996,"kb_used_data":14712,"kb_used_omap":381,"kb_used_meta":3036866,"kb_avail":280063524,"statfs":{"total":289910292480,"available":286785048576,"internally_reserved":0,"allocated":15065088,"data_stored":8181863,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":391062,"internal_metadata":3109750890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-26,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-839418,"num_read_kb":-43245824,"num_write":-1240769,"num_write_kb":-23898365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-82944,"num_keys_recovered":-2,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002042"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.345582+0000","last_change":"2024-05-08T00:30:09.022863+0000","last_active":"2024-05-08T01:01:09.345582+0000","last_peered":"2024-05-08T01:01:09.345582+0000","last_clean":"2024-05-08T01:01:09.345582+0000","last_became_active":"2024-05-08T00:30:09.022284+0000","last_became_peered":"2024-05-08T00:30:09.022284+0000","last_unstale":"2024-05-08T01:01:09.345582+0000","last_undegraded":"2024-05-08T01:01:09.345582+0000","last_fullsized":"2024-05-08T01:01:09.345582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:22:13.046711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.346202+0000","last_change":"2024-05-08T00:30:09.022565+0000","last_active":"2024-05-08T01:01:09.346202+0000","last_peered":"2024-05-08T01:01:09.346202+0000","last_clean":"2024-05-08T01:01:09.346202+0000","last_became_active":"2024-05-08T00:30:09.022263+0000","last_became_peered":"2024-05-08T00:30:09.022263+0000","last_unstale":"2024-05-08T01:01:09.346202+0000","last_undegraded":"2024-05-08T01:01:09.346202+0000","last_fullsized":"2024-05-08T01:01:09.346202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:16:17.936696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347366+0000","last_change":"2024-05-08T00:30:09.022872+0000","last_active":"2024-05-08T01:01:09.347366+0000","last_peered":"2024-05-08T01:01:09.347366+0000","last_clean":"2024-05-08T01:01:09.347366+0000","last_became_active":"2024-05-08T00:30:09.022298+0000","last_became_peered":"2024-05-08T00:30:09.022298+0000","last_unstale":"2024-05-08T01:01:09.347366+0000","last_undegraded":"2024-05-08T01:01:09.347366+0000","last_fullsized":"2024-05-08T01:01:09.347366+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:03:16.834391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347181+0000","last_change":"2024-05-08T00:30:09.022579+0000","last_active":"2024-05-08T01:01:09.347181+0000","last_peered":"2024-05-08T01:01:09.347181+0000","last_clean":"2024-05-08T01:01:09.347181+0000","last_became_active":"2024-05-08T00:30:09.022205+0000","last_became_peered":"2024-05-08T00:30:09.022205+0000","last_unstale":"2024-05-08T01:01:09.347181+0000","last_undegraded":"2024-05-08T01:01:09.347181+0000","last_fullsized":"2024-05-08T01:01:09.347181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:14:39.237433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":835,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:08.676118+0000","last_change":"2024-05-08T00:30:09.023296+0000","last_active":"2024-05-08T01:01:08.676118+0000","last_peered":"2024-05-08T01:01:08.676118+0000","last_clean":"2024-05-08T01:01:08.676118+0000","last_became_active":"2024-05-08T00:30:09.023003+0000","last_became_peered":"2024-05-08T00:30:09.023003+0000","last_unstale":"2024-05-08T01:01:08.676118+0000","last_undegraded":"2024-05-08T01:01:08.676118+0000","last_fullsized":"2024-05-08T01:01:08.676118+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:21:27.320591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":835,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.361667+0000","last_change":"2024-05-08T00:30:09.023325+0000","last_active":"2024-05-08T01:01:09.361667+0000","last_peered":"2024-05-08T01:01:09.361667+0000","last_clean":"2024-05-08T01:01:09.361667+0000","last_became_active":"2024-05-08T00:30:09.022878+0000","last_became_peered":"2024-05-08T00:30:09.022878+0000","last_unstale":"2024-05-08T01:01:09.361667+0000","last_undegraded":"2024-05-08T01:01:09.361667+0000","last_fullsized":"2024-05-08T01:01:09.361667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:18:31.386016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.361976+0000","last_change":"2024-05-08T00:30:09.024010+0000","last_active":"2024-05-08T01:01:09.361976+0000","last_peered":"2024-05-08T01:01:09.361976+0000","last_clean":"2024-05-08T01:01:09.361976+0000","last_became_active":"2024-05-08T00:30:09.023682+0000","last_became_peered":"2024-05-08T00:30:09.023682+0000","last_unstale":"2024-05-08T01:01:09.361976+0000","last_undegraded":"2024-05-08T01:01:09.361976+0000","last_fullsized":"2024-05-08T01:01:09.361976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:07:10.443006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347317+0000","last_change":"2024-05-08T00:30:09.023043+0000","last_active":"2024-05-08T01:01:09.347317+0000","last_peered":"2024-05-08T01:01:09.347317+0000","last_clean":"2024-05-08T01:01:09.347317+0000","last_became_active":"2024-05-08T00:30:09.022369+0000","last_became_peered":"2024-05-08T00:30:09.022369+0000","last_unstale":"2024-05-08T01:01:09.347317+0000","last_undegraded":"2024-05-08T01:01:09.347317+0000","last_fullsized":"2024-05-08T01:01:09.347317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:39:44.848225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":970,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347290+0000","last_change":"2024-05-08T00:30:03.285026+0000","last_active":"2024-05-08T01:01:09.347290+0000","last_peered":"2024-05-08T01:01:09.347290+0000","last_clean":"2024-05-08T01:01:09.347290+0000","last_became_active":"2024-05-08T00:30:03.284046+0000","last_became_peered":"2024-05-08T00:30:03.284046+0000","last_unstale":"2024-05-08T01:01:09.347290+0000","last_undegraded":"2024-05-08T01:01:09.347290+0000","last_fullsized":"2024-05-08T01:01:09.347290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_clean_scrub_stamp":"2024-05-08T00:30:01.996661+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:24:42.092364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738745,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721356,"kb_used_data":4864,"kb_used_omap":132,"kb_used_meta":716347,"kb_avail":93650484,"statfs":{"total":96636764160,"available":95898095616,"internally_reserved":0,"allocated":4980736,"data_stored":2333709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136015,"internal_metadata":733539505},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738746,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1032392,"kb_used_data":4924,"kb_used_omap":111,"kb_used_meta":1027344,"kb_avail":93339448,"statfs":{"total":96636764160,"available":95579594752,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":113964,"internal_metadata":1052000980},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738746,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1298248,"kb_used_data":4924,"kb_used_omap":137,"kb_used_meta":1293174,"kb_avail":93073592,"statfs":{"total":96636764160,"available":95307358208,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":141083,"internal_metadata":1324210405},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:01:14.711 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-05-08T01:01:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T01:01:14.995 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":419,"fsid":"33ffd8b9-c6fe-4232-9b60-cc9dbec54a30","created":"2024-05-08T00:29:52.383902+0000","modified":"2024-05-08T01:01:08.668365+0000","last_up_change":"2024-05-08T00:30:00.964069+0000","last_in_change":"2024-05-08T00:29:53.223004+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T00:30:01.996254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T00:30:07.391983+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3ad08ad1-fedc-43bd-859b-4eb0b0694ac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6809","nonce":2736825466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6811","nonce":2736825466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6815","nonce":2736825466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2736825466},{"type":"v1","addr":"172.21.15.116:6813","nonce":2736825466}]},"public_addr":"172.21.15.116:6809/2736825466","cluster_addr":"172.21.15.116:6811/2736825466","heartbeat_back_addr":"172.21.15.116:6815/2736825466","heartbeat_front_addr":"172.21.15.116:6813/2736825466","state":["exists","up"]},{"osd":1,"uuid":"b27c4c48-7292-4368-adc1-884d8e896c11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6801","nonce":1570588765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6803","nonce":1570588765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6807","nonce":1570588765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1570588765},{"type":"v1","addr":"172.21.15.116:6805","nonce":1570588765}]},"public_addr":"172.21.15.116:6801/1570588765","cluster_addr":"172.21.15.116:6803/1570588765","heartbeat_back_addr":"172.21.15.116:6807/1570588765","heartbeat_front_addr":"172.21.15.116:6805/1570588765","state":["exists","up"]},{"osd":2,"uuid":"98458ca4-a008-4a51-b2e3-14a687146fcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6817","nonce":3557803709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6819","nonce":3557803709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6823","nonce":3557803709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3557803709},{"type":"v1","addr":"172.21.15.116:6821","nonce":3557803709}]},"public_addr":"172.21.15.116:6817/3557803709","cluster_addr":"172.21.15.116:6819/3557803709","heartbeat_back_addr":"172.21.15.116:6823/3557803709","heartbeat_front_addr":"172.21.15.116:6821/3557803709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.326779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.342584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:58.379765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/2239520002":"2024-05-08T01:59:01.496947+0000","172.21.15.116:0/771724031":"2024-05-08T01:58:29.914855+0000","172.21.15.116:0/2609669621":"2024-05-08T01:57:48.103401+0000","172.21.15.116:0/3698936074":"2024-05-08T01:57:03.319460+0000","172.21.15.116:0/2795232399":"2024-05-08T01:54:55.305851+0000","172.21.15.116:0/227166984":"2024-05-08T01:53:07.537131+0000","172.21.15.116:0/3868412757":"2024-05-08T01:49:11.527661+0000","172.21.15.116:0/1628641064":"2024-05-08T01:55:38.782757+0000","172.21.15.116:0/655033290":"2024-05-08T01:49:01.223947+0000","172.21.15.116:0/3025050990":"2024-05-08T01:52:34.177683+0000","172.21.15.116:0/3578814369":"2024-05-08T01:48:41.990748+0000","172.21.15.116:0/3330836984":"2024-05-08T01:47:58.794468+0000","172.21.15.116:0/1321798595":"2024-05-08T01:47:12.561533+0000","172.21.15.116:0/1639163244":"2024-05-08T01:46:01.178464+0000","172.21.15.116:0/710635194":"2024-05-08T01:49:16.750606+0000","172.21.15.116:0/152110175":"2024-05-08T01:47:05.000861+0000","172.21.15.116:0/2667607848":"2024-05-08T01:48:32.910537+0000","172.21.15.116:0/2283670250":"2024-05-08T02:01:03.126720+0000","172.21.15.116:0/1782754272":"2024-05-08T01:46:19.867136+0000","172.21.15.116:0/10775287":"2024-05-08T01:32:46.936567+0000","172.21.15.116:0/3507733580":"2024-05-08T01:50:59.535985+0000","172.21.15.116:0/1968014565":"2024-05-08T01:49:33.490292+0000","172.21.15.116:0/2314120608":"2024-05-08T01:33:35.857410+0000","172.21.15.116:0/1049748652":"2024-05-08T01:52:24.927411+0000","172.21.15.116:0/1204811862":"2024-05-08T01:34:25.286972+0000","172.21.15.116:0/855652211":"2024-05-08T01:39:49.351304+0000","172.21.15.116:0/1637726977":"2024-05-08T01:50:00.783916+0000","172.21.15.116:0/3600836420":"2024-05-08T01:49:38.780436+0000","172.21.15.116:0/2697482136":"2024-05-08T01:49:06.494416+0000","172.21.15.116:0/4238983842":"2024-05-08T01:45:36.723832+0000","172.21.15.116:0/3249890588":"2024-05-08T01:35:31.015303+0000","172.21.15.116:0/1895464404":"2024-05-08T01:48:29.414324+0000","172.21.15.116:0/1546288820":"2024-05-08T01:59:45.395386+0000","172.21.15.116:0/835556711":"2024-05-08T01:36:34.037821+0000","172.21.15.116:0/2321390190":"2024-05-08T01:49:22.647439+0000","172.21.15.116:0/914525907":"2024-05-08T01:46:47.239538+0000","172.21.15.116:0/2410133621":"2024-05-08T01:50:11.948977+0000","172.21.15.116:0/3768761075":"2024-05-08T01:32:05.905250+0000","172.21.15.116:0/462216097":"2024-05-08T01:52:49.530561+0000","172.21.15.116:0/277698606":"2024-05-08T01:38:11.000890+0000","172.21.15.116:0/1274445889":"2024-05-08T01:39:03.107968+0000","172.21.15.116:0/4060167611":"2024-05-08T01:51:45.818942+0000","172.21.15.116:0/4116516889":"2024-05-08T01:50:27.444100+0000","172.21.15.116:0/2030861321":"2024-05-08T01:48:56.084456+0000","172.21.15.116:0/611419151":"2024-05-08T01:37:23.904755+0000","172.21.15.116:0/3482300450":"2024-05-08T01:43:11.867421+0000","172.21.15.116:0/873761794":"2024-05-08T01:49:43.602412+0000","172.21.15.116:0/1847920152":"2024-05-08T01:49:54.600119+0000","172.21.15.116:0/1332691067":"2024-05-08T01:50:07.355171+0000","172.21.15.116:0/3477111375":"2024-05-08T01:51:07.016838+0000","172.21.15.116:0/1865582565":"2024-05-08T02:00:25.475140+0000","172.21.15.116:0/2252548638":"2024-05-08T01:52:29.694157+0000","172.21.15.116:0/1201195784":"2024-05-08T01:50:16.733976+0000","172.21.15.116:0/98220304":"2024-05-08T01:50:21.627474+0000","172.21.15.116:0/2923530974":"2024-05-08T01:52:10.044664+0000","172.21.15.116:0/1549736201":"2024-05-08T01:49:29.032241+0000","172.21.15.116:0/480451621":"2024-05-08T01:52:42.617917+0000","172.21.15.116:0/3972610297":"2024-05-08T01:50:32.936643+0000","172.21.15.116:0/1137023978":"2024-05-08T01:50:38.490499+0000","172.21.15.116:0/1937817251":"2024-05-08T01:56:20.735215+0000","172.21.15.116:0/2058443270":"2024-05-08T01:51:21.606992+0000","172.21.15.116:0/3799084052":"2024-05-08T01:50:49.508366+0000","172.21.15.116:0/2194340168":"2024-05-08T01:50:54.727835+0000","172.21.15.116:0/745204568":"2024-05-08T01:52:00.253881+0000","172.21.15.116:0/1843068300":"2024-05-08T01:51:16.745719+0000","172.21.15.116:0/3019128994":"2024-05-08T01:51:30.780943+0000","172.21.15.116:0/2969658715":"2024-05-08T01:52:19.119123+0000","172.21.15.116:0/1960141414":"2024-05-08T01:50:44.760456+0000","172.21.15.116:0/3478054160":"2024-05-08T01:51:36.865194+0000","172.21.15.116:0/1015752197":"2024-05-08T01:51:50.879289+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T01:01:16.008 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-05-08T01:01:16.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-08T01:01:16.144 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-05-08T01:01:16.145 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T01:01:16.145 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-05-08T01:01:16.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-05-08T01:01:16.442 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 0 to deep-scrub 2024-05-08T01:01:16.456 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-05-08T01:01:16.456 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-08T01:01:16.580 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-05-08T01:01:16.580 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T01:01:16.580 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-05-08T01:01:16.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-05-08T01:01:16.877 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 1 to deep-scrub 2024-05-08T01:01:16.890 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-05-08T01:01:16.890 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-08T01:01:17.013 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-05-08T01:01:17.013 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T01:01:17.013 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-05-08T01:01:17.024 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-05-08T01:01:17.312 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 2 to deep-scrub 2024-05-08T01:01:17.326 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-08T01:01:17.607 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T01:01:17.607 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-05-08T01:01:17.619 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":1355,"stamp":"2024-05-08T01:01:16.418120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3051996,"kb_used_data":14712,"kb_used_omap":381,"kb_used_meta":3036866,"kb_avail":280063524,"statfs":{"total":289910292480,"available":286785048576,"internally_reserved":0,"allocated":15065088,"data_stored":8181863,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":391062,"internal_metadata":3109750890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":-1931,"num_read_kb":-1920,"num_write":-130,"num_write_kb":-127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.747654"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.345582+0000","last_change":"2024-05-08T00:30:09.022863+0000","last_active":"2024-05-08T01:01:09.345582+0000","last_peered":"2024-05-08T01:01:09.345582+0000","last_clean":"2024-05-08T01:01:09.345582+0000","last_became_active":"2024-05-08T00:30:09.022284+0000","last_became_peered":"2024-05-08T00:30:09.022284+0000","last_unstale":"2024-05-08T01:01:09.345582+0000","last_undegraded":"2024-05-08T01:01:09.345582+0000","last_fullsized":"2024-05-08T01:01:09.345582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:22:13.046711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.346202+0000","last_change":"2024-05-08T00:30:09.022565+0000","last_active":"2024-05-08T01:01:09.346202+0000","last_peered":"2024-05-08T01:01:09.346202+0000","last_clean":"2024-05-08T01:01:09.346202+0000","last_became_active":"2024-05-08T00:30:09.022263+0000","last_became_peered":"2024-05-08T00:30:09.022263+0000","last_unstale":"2024-05-08T01:01:09.346202+0000","last_undegraded":"2024-05-08T01:01:09.346202+0000","last_fullsized":"2024-05-08T01:01:09.346202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:16:17.936696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347366+0000","last_change":"2024-05-08T00:30:09.022872+0000","last_active":"2024-05-08T01:01:09.347366+0000","last_peered":"2024-05-08T01:01:09.347366+0000","last_clean":"2024-05-08T01:01:09.347366+0000","last_became_active":"2024-05-08T00:30:09.022298+0000","last_became_peered":"2024-05-08T00:30:09.022298+0000","last_unstale":"2024-05-08T01:01:09.347366+0000","last_undegraded":"2024-05-08T01:01:09.347366+0000","last_fullsized":"2024-05-08T01:01:09.347366+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:03:16.834391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347181+0000","last_change":"2024-05-08T00:30:09.022579+0000","last_active":"2024-05-08T01:01:09.347181+0000","last_peered":"2024-05-08T01:01:09.347181+0000","last_clean":"2024-05-08T01:01:09.347181+0000","last_became_active":"2024-05-08T00:30:09.022205+0000","last_became_peered":"2024-05-08T00:30:09.022205+0000","last_unstale":"2024-05-08T01:01:09.347181+0000","last_undegraded":"2024-05-08T01:01:09.347181+0000","last_fullsized":"2024-05-08T01:01:09.347181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:14:39.237433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":835,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:08.676118+0000","last_change":"2024-05-08T00:30:09.023296+0000","last_active":"2024-05-08T01:01:08.676118+0000","last_peered":"2024-05-08T01:01:08.676118+0000","last_clean":"2024-05-08T01:01:08.676118+0000","last_became_active":"2024-05-08T00:30:09.023003+0000","last_became_peered":"2024-05-08T00:30:09.023003+0000","last_unstale":"2024-05-08T01:01:08.676118+0000","last_undegraded":"2024-05-08T01:01:08.676118+0000","last_fullsized":"2024-05-08T01:01:08.676118+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:21:27.320591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":835,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.361667+0000","last_change":"2024-05-08T00:30:09.023325+0000","last_active":"2024-05-08T01:01:09.361667+0000","last_peered":"2024-05-08T01:01:09.361667+0000","last_clean":"2024-05-08T01:01:09.361667+0000","last_became_active":"2024-05-08T00:30:09.022878+0000","last_became_peered":"2024-05-08T00:30:09.022878+0000","last_unstale":"2024-05-08T01:01:09.361667+0000","last_undegraded":"2024-05-08T01:01:09.361667+0000","last_fullsized":"2024-05-08T01:01:09.361667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:18:31.386016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.361976+0000","last_change":"2024-05-08T00:30:09.024010+0000","last_active":"2024-05-08T01:01:09.361976+0000","last_peered":"2024-05-08T01:01:09.361976+0000","last_clean":"2024-05-08T01:01:09.361976+0000","last_became_active":"2024-05-08T00:30:09.023682+0000","last_became_peered":"2024-05-08T00:30:09.023682+0000","last_unstale":"2024-05-08T01:01:09.361976+0000","last_undegraded":"2024-05-08T01:01:09.361976+0000","last_fullsized":"2024-05-08T01:01:09.361976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:07:10.443006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":836,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347317+0000","last_change":"2024-05-08T00:30:09.023043+0000","last_active":"2024-05-08T01:01:09.347317+0000","last_peered":"2024-05-08T01:01:09.347317+0000","last_clean":"2024-05-08T01:01:09.347317+0000","last_became_active":"2024-05-08T00:30:09.022369+0000","last_became_peered":"2024-05-08T00:30:09.022369+0000","last_unstale":"2024-05-08T01:01:09.347317+0000","last_undegraded":"2024-05-08T01:01:09.347317+0000","last_fullsized":"2024-05-08T01:01:09.347317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:08.008175+0000","last_clean_scrub_stamp":"2024-05-08T00:30:08.008175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:39:44.848225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":970,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:09.347290+0000","last_change":"2024-05-08T00:30:03.285026+0000","last_active":"2024-05-08T01:01:09.347290+0000","last_peered":"2024-05-08T01:01:09.347290+0000","last_clean":"2024-05-08T01:01:09.347290+0000","last_became_active":"2024-05-08T00:30:03.284046+0000","last_became_peered":"2024-05-08T00:30:03.284046+0000","last_unstale":"2024-05-08T01:01:09.347290+0000","last_undegraded":"2024-05-08T01:01:09.347290+0000","last_fullsized":"2024-05-08T01:01:09.347290+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:30:01.996661+0000","last_clean_scrub_stamp":"2024-05-08T00:30:01.996661+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:24:42.092364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721356,"kb_used_data":4864,"kb_used_omap":132,"kb_used_meta":716347,"kb_avail":93650484,"statfs":{"total":96636764160,"available":95898095616,"internally_reserved":0,"allocated":4980736,"data_stored":2333709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136015,"internal_metadata":733539505},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738747,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1032392,"kb_used_data":4924,"kb_used_omap":111,"kb_used_meta":1027344,"kb_avail":93339448,"statfs":{"total":96636764160,"available":95579594752,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":113964,"internal_metadata":1052000980},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738747,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1298248,"kb_used_data":4924,"kb_used_omap":137,"kb_used_meta":1293174,"kb_avail":93073592,"statfs":{"total":96636764160,"available":95307358208,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":141083,"internal_metadata":1324210405},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:01:17.622 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-08T00:30:08.008175+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=30, tm_sec=8, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=15, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:17.622 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-08T00:30:08.008175+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=30, tm_sec=8, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=15, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:17.622 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-08T00:30:08.008175+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=30, tm_sec=8, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=15, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:17.622 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-08T00:30:08.008175+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=30, tm_sec=8, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=15, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:17.622 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-08T00:30:08.008175+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=30, tm_sec=8, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=15, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:17.622 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-08T00:30:08.008175+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=30, tm_sec=8, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=15, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:17.622 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-08T00:30:08.008175+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=30, tm_sec=8, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=15, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:17.623 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-08T00:30:08.008175+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=30, tm_sec=8, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=15, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:17.623 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-08T00:30:01.996661+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=30, tm_sec=1, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=15, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:17.623 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-08T01:01:37.624 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-08T01:01:37.921 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-05-08T01:01:37.921 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-05-08T01:01:37.933 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":1365,"stamp":"2024-05-08T01:01:36.423218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3051984,"kb_used_data":14712,"kb_used_omap":381,"kb_used_meta":3036866,"kb_avail":280063536,"statfs":{"total":289910292480,"available":286785060864,"internally_reserved":0,"allocated":15065088,"data_stored":8181863,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":391062,"internal_metadata":3109750890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002871"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":845,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:20.978807+0000","last_change":"2024-05-08T01:01:20.978830+0000","last_active":"2024-05-08T01:01:20.978807+0000","last_peered":"2024-05-08T01:01:20.978807+0000","last_clean":"2024-05-08T01:01:20.978807+0000","last_became_active":"2024-05-08T00:30:09.022284+0000","last_became_peered":"2024-05-08T00:30:09.022284+0000","last_unstale":"2024-05-08T01:01:20.978807+0000","last_undegraded":"2024-05-08T01:01:20.978807+0000","last_fullsized":"2024-05-08T01:01:20.978807+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:20.978694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:20.978694+0000","last_clean_scrub_stamp":"2024-05-08T01:01:20.978694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:08:39.071024+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":845,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:18.950177+0000","last_change":"2024-05-08T01:01:18.950198+0000","last_active":"2024-05-08T01:01:18.950177+0000","last_peered":"2024-05-08T01:01:18.950177+0000","last_clean":"2024-05-08T01:01:18.950177+0000","last_became_active":"2024-05-08T00:30:09.022263+0000","last_became_peered":"2024-05-08T00:30:09.022263+0000","last_unstale":"2024-05-08T01:01:18.950177+0000","last_undegraded":"2024-05-08T01:01:18.950177+0000","last_fullsized":"2024-05-08T01:01:18.950177+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:18.950076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:18.950076+0000","last_clean_scrub_stamp":"2024-05-08T01:01:18.950076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:53:54.739035+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":846,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:21.945061+0000","last_change":"2024-05-08T01:01:21.945061+0000","last_active":"2024-05-08T01:01:21.945061+0000","last_peered":"2024-05-08T01:01:21.945061+0000","last_clean":"2024-05-08T01:01:21.945061+0000","last_became_active":"2024-05-08T00:30:09.022298+0000","last_became_peered":"2024-05-08T00:30:09.022298+0000","last_unstale":"2024-05-08T01:01:21.945061+0000","last_undegraded":"2024-05-08T01:01:21.945061+0000","last_fullsized":"2024-05-08T01:01:21.945061+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:21.944909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:21.944909+0000","last_clean_scrub_stamp":"2024-05-08T01:01:21.944909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:13:54.362006+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":846,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:19.892504+0000","last_change":"2024-05-08T01:01:19.892504+0000","last_active":"2024-05-08T01:01:19.892504+0000","last_peered":"2024-05-08T01:01:19.892504+0000","last_clean":"2024-05-08T01:01:19.892504+0000","last_became_active":"2024-05-08T00:30:09.022205+0000","last_became_peered":"2024-05-08T00:30:09.022205+0000","last_unstale":"2024-05-08T01:01:19.892504+0000","last_undegraded":"2024-05-08T01:01:19.892504+0000","last_fullsized":"2024-05-08T01:01:19.892504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:19.892366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:19.892366+0000","last_clean_scrub_stamp":"2024-05-08T01:01:19.892366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:58:28.957554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":845,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:17.130716+0000","last_change":"2024-05-08T01:01:17.130716+0000","last_active":"2024-05-08T01:01:17.130716+0000","last_peered":"2024-05-08T01:01:17.130716+0000","last_clean":"2024-05-08T01:01:17.130716+0000","last_became_active":"2024-05-08T00:30:09.023003+0000","last_became_peered":"2024-05-08T00:30:09.023003+0000","last_unstale":"2024-05-08T01:01:17.130716+0000","last_undegraded":"2024-05-08T01:01:17.130716+0000","last_fullsized":"2024-05-08T01:01:17.130716+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:17.130599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:17.130599+0000","last_clean_scrub_stamp":"2024-05-08T01:01:17.130599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:04:14.689027+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":845,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:17.927831+0000","last_change":"2024-05-08T01:01:17.927831+0000","last_active":"2024-05-08T01:01:17.927831+0000","last_peered":"2024-05-08T01:01:17.927831+0000","last_clean":"2024-05-08T01:01:17.927831+0000","last_became_active":"2024-05-08T00:30:09.022878+0000","last_became_peered":"2024-05-08T00:30:09.022878+0000","last_unstale":"2024-05-08T01:01:17.927831+0000","last_undegraded":"2024-05-08T01:01:17.927831+0000","last_fullsized":"2024-05-08T01:01:17.927831+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:17.927693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:17.927693+0000","last_clean_scrub_stamp":"2024-05-08T01:01:17.927693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:22:08.959767+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":845,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:18.906415+0000","last_change":"2024-05-08T01:01:18.906447+0000","last_active":"2024-05-08T01:01:18.906415+0000","last_peered":"2024-05-08T01:01:18.906415+0000","last_clean":"2024-05-08T01:01:18.906415+0000","last_became_active":"2024-05-08T00:30:09.023682+0000","last_became_peered":"2024-05-08T00:30:09.023682+0000","last_unstale":"2024-05-08T01:01:18.906415+0000","last_undegraded":"2024-05-08T01:01:18.906415+0000","last_fullsized":"2024-05-08T01:01:18.906415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:18.906277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:18.906277+0000","last_clean_scrub_stamp":"2024-05-08T01:01:18.906277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:02:58.521217+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":846,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:22.986151+0000","last_change":"2024-05-08T01:01:22.986151+0000","last_active":"2024-05-08T01:01:22.986151+0000","last_peered":"2024-05-08T01:01:22.986151+0000","last_clean":"2024-05-08T01:01:22.986151+0000","last_became_active":"2024-05-08T00:30:09.022369+0000","last_became_peered":"2024-05-08T00:30:09.022369+0000","last_unstale":"2024-05-08T01:01:22.986151+0000","last_undegraded":"2024-05-08T01:01:22.986151+0000","last_fullsized":"2024-05-08T01:01:22.986151+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:22.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:22.986005+0000","last_clean_scrub_stamp":"2024-05-08T01:01:22.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T12:07:32.366653+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"419'99","reported_seq":981,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-05-08T01:01:17.898827+0000","last_change":"2024-05-08T01:01:17.898857+0000","last_active":"2024-05-08T01:01:17.898827+0000","last_peered":"2024-05-08T01:01:17.898827+0000","last_clean":"2024-05-08T01:01:17.898827+0000","last_became_active":"2024-05-08T00:30:03.284046+0000","last_became_peered":"2024-05-08T00:30:03.284046+0000","last_unstale":"2024-05-08T01:01:17.898827+0000","last_undegraded":"2024-05-08T01:01:17.898827+0000","last_fullsized":"2024-05-08T01:01:17.898827+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'99","last_scrub_stamp":"2024-05-08T01:01:17.898757+0000","last_deep_scrub":"419'99","last_deep_scrub_stamp":"2024-05-08T01:01:17.898757+0000","last_clean_scrub_stamp":"2024-05-08T01:01:17.898757+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T09:09:02.156757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59904,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738750,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721352,"kb_used_data":4864,"kb_used_omap":132,"kb_used_meta":716347,"kb_avail":93650488,"statfs":{"total":96636764160,"available":95898099712,"internally_reserved":0,"allocated":4980736,"data_stored":2333709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136015,"internal_metadata":733539505},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738751,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1032388,"kb_used_data":4924,"kb_used_omap":111,"kb_used_meta":1027344,"kb_avail":93339452,"statfs":{"total":96636764160,"available":95579598848,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":113964,"internal_metadata":1052000980},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738751,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1298244,"kb_used_data":4924,"kb_used_omap":137,"kb_used_meta":1293174,"kb_avail":93073596,"statfs":{"total":96636764160,"available":95307362304,"internally_reserved":0,"allocated":5042176,"data_stored":2924077,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":141083,"internal_metadata":1324210405},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29952,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:01:37.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-05-08T01:01:38.233 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-08T01:01:38.233 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-08T01:01:38.234 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-05-08T01:01:38.234 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:01:38.300 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-05-08T01:01:38.300 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-05-08T01:01:38.300 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:01:38.368 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-05-08T01:01:38.368 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-05-08T01:01:38.368 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:01:38.430 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-05-08T01:01:38.430 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-08T01:01:38.430 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-05-08T01:01:38.430 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:01:38.459 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-05-08T01:01:38.459 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-08T01:01:38.459 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-05-08T01:01:38.459 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:01:38.472 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-05-08T01:01:38.472 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-08T01:01:38.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-05-08T01:01:38.600 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T01:01:38.600 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-05-08T01:01:38.997 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T01:01:38.998 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-05-08T01:01:39.141 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T01:01:39.141 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-05-08T01:01:39.272 INFO:tasks.ceph:Archiving mon data... 2024-05-08T01:01:39.273 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630/data/cluster2.mon.a.tgz 2024-05-08T01:01:39.273 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-05-08T01:01:39.288 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.04F4GgeqOV 2024-05-08T01:01:39.289 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.04F4GgeqOV 2024-05-08T01:01:39.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.04F4GgeqOV 2024-05-08T01:01:39.562 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.04F4GgeqOV is 480KB 2024-05-08T01:01:39.661 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.04F4GgeqOV 2024-05-08T01:01:39.676 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-08T01:01:39.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-05-08T01:01:39.679 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-05-08T01:01:39.754 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-08T01:01:39.765 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-05-08T01:01:39.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-08T01:01:40.327 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T01:01:40.327 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-05-08T01:01:40.339 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1300,"stamp":"2024-05-08T01:01:38.423618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4406972,"kb_used_data":8208,"kb_used_omap":473,"kb_used_meta":4398246,"kb_avail":278708548,"statfs":{"total":289910292480,"available":285397553152,"internally_reserved":0,"allocated":8404992,"data_stored":5561924,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":485253,"internal_metadata":4503804027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.684383+0000","last_change":"2024-05-08T00:29:32.298717+0000","last_active":"2024-05-08T01:01:07.684383+0000","last_peered":"2024-05-08T01:01:07.684383+0000","last_clean":"2024-05-08T01:01:07.684383+0000","last_became_active":"2024-05-08T00:29:32.298235+0000","last_became_peered":"2024-05-08T00:29:32.298235+0000","last_unstale":"2024-05-08T01:01:07.684383+0000","last_undegraded":"2024-05-08T01:01:07.684383+0000","last_fullsized":"2024-05-08T01:01:07.684383+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:29:00.040409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.683825+0000","last_change":"2024-05-08T00:29:32.297238+0000","last_active":"2024-05-08T01:01:07.683825+0000","last_peered":"2024-05-08T01:01:07.683825+0000","last_clean":"2024-05-08T01:01:07.683825+0000","last_became_active":"2024-05-08T00:29:32.296922+0000","last_became_peered":"2024-05-08T00:29:32.296922+0000","last_unstale":"2024-05-08T01:01:07.683825+0000","last_undegraded":"2024-05-08T01:01:07.683825+0000","last_fullsized":"2024-05-08T01:01:07.683825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:35:49.661682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.682806+0000","last_change":"2024-05-08T00:29:32.296077+0000","last_active":"2024-05-08T01:01:07.682806+0000","last_peered":"2024-05-08T01:01:07.682806+0000","last_clean":"2024-05-08T01:01:07.682806+0000","last_became_active":"2024-05-08T00:29:32.295815+0000","last_became_peered":"2024-05-08T00:29:32.295815+0000","last_unstale":"2024-05-08T01:01:07.682806+0000","last_undegraded":"2024-05-08T01:01:07.682806+0000","last_fullsized":"2024-05-08T01:01:07.682806+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:48.464488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.680927+0000","last_change":"2024-05-08T00:29:32.298850+0000","last_active":"2024-05-08T01:01:07.680927+0000","last_peered":"2024-05-08T01:01:07.680927+0000","last_clean":"2024-05-08T01:01:07.680927+0000","last_became_active":"2024-05-08T00:29:32.298232+0000","last_became_peered":"2024-05-08T00:29:32.298232+0000","last_unstale":"2024-05-08T01:01:07.680927+0000","last_undegraded":"2024-05-08T01:01:07.680927+0000","last_fullsized":"2024-05-08T01:01:07.680927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:37:54.763429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":667,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.700664+0000","last_change":"2024-05-08T00:29:32.297972+0000","last_active":"2024-05-08T01:01:07.700664+0000","last_peered":"2024-05-08T01:01:07.700664+0000","last_clean":"2024-05-08T01:01:07.700664+0000","last_became_active":"2024-05-08T00:29:32.297691+0000","last_became_peered":"2024-05-08T00:29:32.297691+0000","last_unstale":"2024-05-08T01:01:07.700664+0000","last_undegraded":"2024-05-08T01:01:07.700664+0000","last_fullsized":"2024-05-08T01:01:07.700664+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:34:10.411554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.693670+0000","last_change":"2024-05-08T00:29:32.295667+0000","last_active":"2024-05-08T01:01:07.693670+0000","last_peered":"2024-05-08T01:01:07.693670+0000","last_clean":"2024-05-08T01:01:07.693670+0000","last_became_active":"2024-05-08T00:29:32.295343+0000","last_became_peered":"2024-05-08T00:29:32.295343+0000","last_unstale":"2024-05-08T01:01:07.693670+0000","last_undegraded":"2024-05-08T01:01:07.693670+0000","last_fullsized":"2024-05-08T01:01:07.693670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:15:37.989200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.693927+0000","last_change":"2024-05-08T00:29:32.297596+0000","last_active":"2024-05-08T01:01:07.693927+0000","last_peered":"2024-05-08T01:01:07.693927+0000","last_clean":"2024-05-08T01:01:07.693927+0000","last_became_active":"2024-05-08T00:29:32.297247+0000","last_became_peered":"2024-05-08T00:29:32.297247+0000","last_unstale":"2024-05-08T01:01:07.693927+0000","last_undegraded":"2024-05-08T01:01:07.693927+0000","last_fullsized":"2024-05-08T01:01:07.693927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:12:16.421814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.682625+0000","last_change":"2024-05-08T00:29:32.295731+0000","last_active":"2024-05-08T01:01:07.682625+0000","last_peered":"2024-05-08T01:01:07.682625+0000","last_clean":"2024-05-08T01:01:07.682625+0000","last_became_active":"2024-05-08T00:29:32.295380+0000","last_became_peered":"2024-05-08T00:29:32.295380+0000","last_unstale":"2024-05-08T01:01:07.682625+0000","last_undegraded":"2024-05-08T01:01:07.682625+0000","last_fullsized":"2024-05-08T01:01:07.682625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:19:18.847185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":800,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.678212+0000","last_change":"2024-05-08T00:29:27.459396+0000","last_active":"2024-05-08T01:01:07.678212+0000","last_peered":"2024-05-08T01:01:07.678212+0000","last_clean":"2024-05-08T01:01:07.678212+0000","last_became_active":"2024-05-08T00:29:27.458282+0000","last_became_peered":"2024-05-08T00:29:27.458282+0000","last_unstale":"2024-05-08T01:01:07.678212+0000","last_undegraded":"2024-05-08T01:01:07.678212+0000","last_fullsized":"2024-05-08T01:01:07.678212+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_clean_scrub_stamp":"2024-05-08T00:29:26.256116+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:00:43.917924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738758,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801172,"kb_used_data":2696,"kb_used_omap":118,"kb_used_meta":1798345,"kb_avail":92570668,"statfs":{"total":96636764160,"available":94792364032,"internally_reserved":0,"allocated":2760704,"data_stored":1460396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121245,"internal_metadata":1841505891},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738759,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1421780,"kb_used_data":2756,"kb_used_omap":161,"kb_used_meta":1418846,"kb_avail":92950060,"statfs":{"total":96636764160,"available":95180861440,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":165279,"internal_metadata":1452898913},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738757,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1184020,"kb_used_data":2756,"kb_used_omap":194,"kb_used_meta":1181053,"kb_avail":93187820,"statfs":{"total":96636764160,"available":95424327680,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":198729,"internal_metadata":1209399223},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:01:40.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-08T01:01:40.637 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T01:01:40.638 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-05-08T01:01:40.649 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1301,"stamp":"2024-05-08T01:01:40.423949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4406972,"kb_used_data":8208,"kb_used_omap":473,"kb_used_meta":4398246,"kb_avail":278708548,"statfs":{"total":289910292480,"available":285397553152,"internally_reserved":0,"allocated":8404992,"data_stored":5561924,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":485253,"internal_metadata":4503804027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002556"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.684383+0000","last_change":"2024-05-08T00:29:32.298717+0000","last_active":"2024-05-08T01:01:07.684383+0000","last_peered":"2024-05-08T01:01:07.684383+0000","last_clean":"2024-05-08T01:01:07.684383+0000","last_became_active":"2024-05-08T00:29:32.298235+0000","last_became_peered":"2024-05-08T00:29:32.298235+0000","last_unstale":"2024-05-08T01:01:07.684383+0000","last_undegraded":"2024-05-08T01:01:07.684383+0000","last_fullsized":"2024-05-08T01:01:07.684383+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:29:00.040409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.683825+0000","last_change":"2024-05-08T00:29:32.297238+0000","last_active":"2024-05-08T01:01:07.683825+0000","last_peered":"2024-05-08T01:01:07.683825+0000","last_clean":"2024-05-08T01:01:07.683825+0000","last_became_active":"2024-05-08T00:29:32.296922+0000","last_became_peered":"2024-05-08T00:29:32.296922+0000","last_unstale":"2024-05-08T01:01:07.683825+0000","last_undegraded":"2024-05-08T01:01:07.683825+0000","last_fullsized":"2024-05-08T01:01:07.683825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:35:49.661682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.682806+0000","last_change":"2024-05-08T00:29:32.296077+0000","last_active":"2024-05-08T01:01:07.682806+0000","last_peered":"2024-05-08T01:01:07.682806+0000","last_clean":"2024-05-08T01:01:07.682806+0000","last_became_active":"2024-05-08T00:29:32.295815+0000","last_became_peered":"2024-05-08T00:29:32.295815+0000","last_unstale":"2024-05-08T01:01:07.682806+0000","last_undegraded":"2024-05-08T01:01:07.682806+0000","last_fullsized":"2024-05-08T01:01:07.682806+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:48.464488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.680927+0000","last_change":"2024-05-08T00:29:32.298850+0000","last_active":"2024-05-08T01:01:07.680927+0000","last_peered":"2024-05-08T01:01:07.680927+0000","last_clean":"2024-05-08T01:01:07.680927+0000","last_became_active":"2024-05-08T00:29:32.298232+0000","last_became_peered":"2024-05-08T00:29:32.298232+0000","last_unstale":"2024-05-08T01:01:07.680927+0000","last_undegraded":"2024-05-08T01:01:07.680927+0000","last_fullsized":"2024-05-08T01:01:07.680927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:37:54.763429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":667,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.700664+0000","last_change":"2024-05-08T00:29:32.297972+0000","last_active":"2024-05-08T01:01:07.700664+0000","last_peered":"2024-05-08T01:01:07.700664+0000","last_clean":"2024-05-08T01:01:07.700664+0000","last_became_active":"2024-05-08T00:29:32.297691+0000","last_became_peered":"2024-05-08T00:29:32.297691+0000","last_unstale":"2024-05-08T01:01:07.700664+0000","last_undegraded":"2024-05-08T01:01:07.700664+0000","last_fullsized":"2024-05-08T01:01:07.700664+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:34:10.411554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.693670+0000","last_change":"2024-05-08T00:29:32.295667+0000","last_active":"2024-05-08T01:01:07.693670+0000","last_peered":"2024-05-08T01:01:07.693670+0000","last_clean":"2024-05-08T01:01:07.693670+0000","last_became_active":"2024-05-08T00:29:32.295343+0000","last_became_peered":"2024-05-08T00:29:32.295343+0000","last_unstale":"2024-05-08T01:01:07.693670+0000","last_undegraded":"2024-05-08T01:01:07.693670+0000","last_fullsized":"2024-05-08T01:01:07.693670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:15:37.989200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.693927+0000","last_change":"2024-05-08T00:29:32.297596+0000","last_active":"2024-05-08T01:01:07.693927+0000","last_peered":"2024-05-08T01:01:07.693927+0000","last_clean":"2024-05-08T01:01:07.693927+0000","last_became_active":"2024-05-08T00:29:32.297247+0000","last_became_peered":"2024-05-08T00:29:32.297247+0000","last_unstale":"2024-05-08T01:01:07.693927+0000","last_undegraded":"2024-05-08T01:01:07.693927+0000","last_fullsized":"2024-05-08T01:01:07.693927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:12:16.421814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.682625+0000","last_change":"2024-05-08T00:29:32.295731+0000","last_active":"2024-05-08T01:01:07.682625+0000","last_peered":"2024-05-08T01:01:07.682625+0000","last_clean":"2024-05-08T01:01:07.682625+0000","last_became_active":"2024-05-08T00:29:32.295380+0000","last_became_peered":"2024-05-08T00:29:32.295380+0000","last_unstale":"2024-05-08T01:01:07.682625+0000","last_undegraded":"2024-05-08T01:01:07.682625+0000","last_fullsized":"2024-05-08T01:01:07.682625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:19:18.847185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":800,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.678212+0000","last_change":"2024-05-08T00:29:27.459396+0000","last_active":"2024-05-08T01:01:07.678212+0000","last_peered":"2024-05-08T01:01:07.678212+0000","last_clean":"2024-05-08T01:01:07.678212+0000","last_became_active":"2024-05-08T00:29:27.458282+0000","last_became_peered":"2024-05-08T00:29:27.458282+0000","last_unstale":"2024-05-08T01:01:07.678212+0000","last_undegraded":"2024-05-08T01:01:07.678212+0000","last_fullsized":"2024-05-08T01:01:07.678212+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_clean_scrub_stamp":"2024-05-08T00:29:26.256116+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:00:43.917924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738758,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801172,"kb_used_data":2696,"kb_used_omap":118,"kb_used_meta":1798345,"kb_avail":92570668,"statfs":{"total":96636764160,"available":94792364032,"internally_reserved":0,"allocated":2760704,"data_stored":1460396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121245,"internal_metadata":1841505891},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738759,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1421780,"kb_used_data":2756,"kb_used_omap":161,"kb_used_meta":1418846,"kb_avail":92950060,"statfs":{"total":96636764160,"available":95180861440,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":165279,"internal_metadata":1452898913},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738757,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1184020,"kb_used_data":2756,"kb_used_omap":194,"kb_used_meta":1181053,"kb_avail":93187820,"statfs":{"total":96636764160,"available":95424327680,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":198729,"internal_metadata":1209399223},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:01:40.650 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-05-08T01:01:40.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-08T01:01:40.948 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T01:01:40.949 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-05-08T01:01:40.961 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1301,"stamp":"2024-05-08T01:01:40.423949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4406972,"kb_used_data":8208,"kb_used_omap":473,"kb_used_meta":4398246,"kb_avail":278708548,"statfs":{"total":289910292480,"available":285397553152,"internally_reserved":0,"allocated":8404992,"data_stored":5561924,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":485253,"internal_metadata":4503804027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002556"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.684383+0000","last_change":"2024-05-08T00:29:32.298717+0000","last_active":"2024-05-08T01:01:07.684383+0000","last_peered":"2024-05-08T01:01:07.684383+0000","last_clean":"2024-05-08T01:01:07.684383+0000","last_became_active":"2024-05-08T00:29:32.298235+0000","last_became_peered":"2024-05-08T00:29:32.298235+0000","last_unstale":"2024-05-08T01:01:07.684383+0000","last_undegraded":"2024-05-08T01:01:07.684383+0000","last_fullsized":"2024-05-08T01:01:07.684383+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:29:00.040409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.683825+0000","last_change":"2024-05-08T00:29:32.297238+0000","last_active":"2024-05-08T01:01:07.683825+0000","last_peered":"2024-05-08T01:01:07.683825+0000","last_clean":"2024-05-08T01:01:07.683825+0000","last_became_active":"2024-05-08T00:29:32.296922+0000","last_became_peered":"2024-05-08T00:29:32.296922+0000","last_unstale":"2024-05-08T01:01:07.683825+0000","last_undegraded":"2024-05-08T01:01:07.683825+0000","last_fullsized":"2024-05-08T01:01:07.683825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:35:49.661682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.682806+0000","last_change":"2024-05-08T00:29:32.296077+0000","last_active":"2024-05-08T01:01:07.682806+0000","last_peered":"2024-05-08T01:01:07.682806+0000","last_clean":"2024-05-08T01:01:07.682806+0000","last_became_active":"2024-05-08T00:29:32.295815+0000","last_became_peered":"2024-05-08T00:29:32.295815+0000","last_unstale":"2024-05-08T01:01:07.682806+0000","last_undegraded":"2024-05-08T01:01:07.682806+0000","last_fullsized":"2024-05-08T01:01:07.682806+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:48.464488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.680927+0000","last_change":"2024-05-08T00:29:32.298850+0000","last_active":"2024-05-08T01:01:07.680927+0000","last_peered":"2024-05-08T01:01:07.680927+0000","last_clean":"2024-05-08T01:01:07.680927+0000","last_became_active":"2024-05-08T00:29:32.298232+0000","last_became_peered":"2024-05-08T00:29:32.298232+0000","last_unstale":"2024-05-08T01:01:07.680927+0000","last_undegraded":"2024-05-08T01:01:07.680927+0000","last_fullsized":"2024-05-08T01:01:07.680927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:37:54.763429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":667,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.700664+0000","last_change":"2024-05-08T00:29:32.297972+0000","last_active":"2024-05-08T01:01:07.700664+0000","last_peered":"2024-05-08T01:01:07.700664+0000","last_clean":"2024-05-08T01:01:07.700664+0000","last_became_active":"2024-05-08T00:29:32.297691+0000","last_became_peered":"2024-05-08T00:29:32.297691+0000","last_unstale":"2024-05-08T01:01:07.700664+0000","last_undegraded":"2024-05-08T01:01:07.700664+0000","last_fullsized":"2024-05-08T01:01:07.700664+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:34:10.411554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.693670+0000","last_change":"2024-05-08T00:29:32.295667+0000","last_active":"2024-05-08T01:01:07.693670+0000","last_peered":"2024-05-08T01:01:07.693670+0000","last_clean":"2024-05-08T01:01:07.693670+0000","last_became_active":"2024-05-08T00:29:32.295343+0000","last_became_peered":"2024-05-08T00:29:32.295343+0000","last_unstale":"2024-05-08T01:01:07.693670+0000","last_undegraded":"2024-05-08T01:01:07.693670+0000","last_fullsized":"2024-05-08T01:01:07.693670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:15:37.989200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.693927+0000","last_change":"2024-05-08T00:29:32.297596+0000","last_active":"2024-05-08T01:01:07.693927+0000","last_peered":"2024-05-08T01:01:07.693927+0000","last_clean":"2024-05-08T01:01:07.693927+0000","last_became_active":"2024-05-08T00:29:32.297247+0000","last_became_peered":"2024-05-08T00:29:32.297247+0000","last_unstale":"2024-05-08T01:01:07.693927+0000","last_undegraded":"2024-05-08T01:01:07.693927+0000","last_fullsized":"2024-05-08T01:01:07.693927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:12:16.421814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.682625+0000","last_change":"2024-05-08T00:29:32.295731+0000","last_active":"2024-05-08T01:01:07.682625+0000","last_peered":"2024-05-08T01:01:07.682625+0000","last_clean":"2024-05-08T01:01:07.682625+0000","last_became_active":"2024-05-08T00:29:32.295380+0000","last_became_peered":"2024-05-08T00:29:32.295380+0000","last_unstale":"2024-05-08T01:01:07.682625+0000","last_undegraded":"2024-05-08T01:01:07.682625+0000","last_fullsized":"2024-05-08T01:01:07.682625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:19:18.847185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":800,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.678212+0000","last_change":"2024-05-08T00:29:27.459396+0000","last_active":"2024-05-08T01:01:07.678212+0000","last_peered":"2024-05-08T01:01:07.678212+0000","last_clean":"2024-05-08T01:01:07.678212+0000","last_became_active":"2024-05-08T00:29:27.458282+0000","last_became_peered":"2024-05-08T00:29:27.458282+0000","last_unstale":"2024-05-08T01:01:07.678212+0000","last_undegraded":"2024-05-08T01:01:07.678212+0000","last_fullsized":"2024-05-08T01:01:07.678212+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_clean_scrub_stamp":"2024-05-08T00:29:26.256116+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:00:43.917924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738758,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801172,"kb_used_data":2696,"kb_used_omap":118,"kb_used_meta":1798345,"kb_avail":92570668,"statfs":{"total":96636764160,"available":94792364032,"internally_reserved":0,"allocated":2760704,"data_stored":1460396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121245,"internal_metadata":1841505891},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738759,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1421780,"kb_used_data":2756,"kb_used_omap":161,"kb_used_meta":1418846,"kb_avail":92950060,"statfs":{"total":96636764160,"available":95180861440,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":165279,"internal_metadata":1452898913},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738757,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1184020,"kb_used_data":2756,"kb_used_omap":194,"kb_used_meta":1181053,"kb_avail":93187820,"statfs":{"total":96636764160,"available":95424327680,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":198729,"internal_metadata":1209399223},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:01:40.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-05-08T01:01:41.265 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T01:01:41.266 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":335,"fsid":"4f2229d1-1d9b-4660-ad3b-ce8a6d783878","created":"2024-05-08T00:29:15.407366+0000","modified":"2024-05-08T01:01:07.673098+0000","last_up_change":"2024-05-08T00:29:24.251757+0000","last_in_change":"2024-05-08T00:29:16.330023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-08T00:29:26.220314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-08T00:29:30.582717+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"66871eb8-7afb-4542-9b36-201bfb221deb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6817","nonce":1089770871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6819","nonce":1089770871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6823","nonce":1089770871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":1089770871},{"type":"v1","addr":"172.21.15.64:6821","nonce":1089770871}]},"public_addr":"172.21.15.64:6817/1089770871","cluster_addr":"172.21.15.64:6819/1089770871","heartbeat_back_addr":"172.21.15.64:6823/1089770871","heartbeat_front_addr":"172.21.15.64:6821/1089770871","state":["exists","up"]},{"osd":1,"uuid":"d13ef93c-d6a7-46a7-8190-3c39b06d43d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6801","nonce":3957903691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6803","nonce":3957903691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6807","nonce":3957903691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3957903691},{"type":"v1","addr":"172.21.15.64:6805","nonce":3957903691}]},"public_addr":"172.21.15.64:6801/3957903691","cluster_addr":"172.21.15.64:6803/3957903691","heartbeat_back_addr":"172.21.15.64:6807/3957903691","heartbeat_front_addr":"172.21.15.64:6805/3957903691","state":["exists","up"]},{"osd":2,"uuid":"35fef95e-5bb0-4022-8a3a-4b4d9ee78eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6809","nonce":467041737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6811","nonce":467041737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6815","nonce":467041737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":467041737},{"type":"v1","addr":"172.21.15.64:6813","nonce":467041737}]},"public_addr":"172.21.15.64:6809/467041737","cluster_addr":"172.21.15.64:6811/467041737","heartbeat_back_addr":"172.21.15.64:6815/467041737","heartbeat_front_addr":"172.21.15.64:6813/467041737","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.522965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.442692+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T00:29:21.461612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T01:01:42.280 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-05-08T01:01:42.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-08T01:01:42.424 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-05-08T01:01:42.424 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T01:01:42.424 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-05-08T01:01:42.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-05-08T01:01:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 0 to deep-scrub 2024-05-08T01:01:42.747 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-05-08T01:01:42.747 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-08T01:01:42.876 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-05-08T01:01:42.876 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T01:01:42.876 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-05-08T01:01:42.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-05-08T01:01:43.188 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 1 to deep-scrub 2024-05-08T01:01:43.201 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-05-08T01:01:43.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-08T01:01:43.329 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-05-08T01:01:43.329 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T01:01:43.329 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-05-08T01:01:43.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-05-08T01:01:43.639 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 2 to deep-scrub 2024-05-08T01:01:43.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-08T01:01:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T01:01:43.950 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-05-08T01:01:43.963 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1302,"stamp":"2024-05-08T01:01:42.424198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4406972,"kb_used_data":8208,"kb_used_omap":473,"kb_used_meta":4398246,"kb_avail":278708548,"statfs":{"total":289910292480,"available":285397553152,"internally_reserved":0,"allocated":8404992,"data_stored":5561924,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":485253,"internal_metadata":4503804027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002352"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.684383+0000","last_change":"2024-05-08T00:29:32.298717+0000","last_active":"2024-05-08T01:01:07.684383+0000","last_peered":"2024-05-08T01:01:07.684383+0000","last_clean":"2024-05-08T01:01:07.684383+0000","last_became_active":"2024-05-08T00:29:32.298235+0000","last_became_peered":"2024-05-08T00:29:32.298235+0000","last_unstale":"2024-05-08T01:01:07.684383+0000","last_undegraded":"2024-05-08T01:01:07.684383+0000","last_fullsized":"2024-05-08T01:01:07.684383+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T08:29:00.040409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.683825+0000","last_change":"2024-05-08T00:29:32.297238+0000","last_active":"2024-05-08T01:01:07.683825+0000","last_peered":"2024-05-08T01:01:07.683825+0000","last_clean":"2024-05-08T01:01:07.683825+0000","last_became_active":"2024-05-08T00:29:32.296922+0000","last_became_peered":"2024-05-08T00:29:32.296922+0000","last_unstale":"2024-05-08T01:01:07.683825+0000","last_undegraded":"2024-05-08T01:01:07.683825+0000","last_fullsized":"2024-05-08T01:01:07.683825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:35:49.661682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.682806+0000","last_change":"2024-05-08T00:29:32.296077+0000","last_active":"2024-05-08T01:01:07.682806+0000","last_peered":"2024-05-08T01:01:07.682806+0000","last_clean":"2024-05-08T01:01:07.682806+0000","last_became_active":"2024-05-08T00:29:32.295815+0000","last_became_peered":"2024-05-08T00:29:32.295815+0000","last_unstale":"2024-05-08T01:01:07.682806+0000","last_undegraded":"2024-05-08T01:01:07.682806+0000","last_fullsized":"2024-05-08T01:01:07.682806+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:50:48.464488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.680927+0000","last_change":"2024-05-08T00:29:32.298850+0000","last_active":"2024-05-08T01:01:07.680927+0000","last_peered":"2024-05-08T01:01:07.680927+0000","last_clean":"2024-05-08T01:01:07.680927+0000","last_became_active":"2024-05-08T00:29:32.298232+0000","last_became_peered":"2024-05-08T00:29:32.298232+0000","last_unstale":"2024-05-08T01:01:07.680927+0000","last_undegraded":"2024-05-08T01:01:07.680927+0000","last_fullsized":"2024-05-08T01:01:07.680927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:37:54.763429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":667,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.700664+0000","last_change":"2024-05-08T00:29:32.297972+0000","last_active":"2024-05-08T01:01:07.700664+0000","last_peered":"2024-05-08T01:01:07.700664+0000","last_clean":"2024-05-08T01:01:07.700664+0000","last_became_active":"2024-05-08T00:29:32.297691+0000","last_became_peered":"2024-05-08T00:29:32.297691+0000","last_unstale":"2024-05-08T01:01:07.700664+0000","last_undegraded":"2024-05-08T01:01:07.700664+0000","last_fullsized":"2024-05-08T01:01:07.700664+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:34:10.411554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":665,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.693670+0000","last_change":"2024-05-08T00:29:32.295667+0000","last_active":"2024-05-08T01:01:07.693670+0000","last_peered":"2024-05-08T01:01:07.693670+0000","last_clean":"2024-05-08T01:01:07.693670+0000","last_became_active":"2024-05-08T00:29:32.295343+0000","last_became_peered":"2024-05-08T00:29:32.295343+0000","last_unstale":"2024-05-08T01:01:07.693670+0000","last_undegraded":"2024-05-08T01:01:07.693670+0000","last_fullsized":"2024-05-08T01:01:07.693670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:15:37.989200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.693953+0000","last_change":"2024-05-08T00:29:32.297596+0000","last_active":"2024-05-08T01:01:07.693953+0000","last_peered":"2024-05-08T01:01:07.693953+0000","last_clean":"2024-05-08T01:01:07.693953+0000","last_became_active":"2024-05-08T00:29:32.297247+0000","last_became_peered":"2024-05-08T00:29:32.297247+0000","last_unstale":"2024-05-08T01:01:07.693953+0000","last_undegraded":"2024-05-08T01:01:07.693953+0000","last_fullsized":"2024-05-08T01:01:07.693953+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T01:12:16.421814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.682625+0000","last_change":"2024-05-08T00:29:32.295731+0000","last_active":"2024-05-08T01:01:07.682625+0000","last_peered":"2024-05-08T01:01:07.682625+0000","last_clean":"2024-05-08T01:01:07.682625+0000","last_became_active":"2024-05-08T00:29:32.295380+0000","last_became_peered":"2024-05-08T00:29:32.295380+0000","last_unstale":"2024-05-08T01:01:07.682625+0000","last_undegraded":"2024-05-08T01:01:07.682625+0000","last_fullsized":"2024-05-08T01:01:07.682625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:31.271635+0000","last_clean_scrub_stamp":"2024-05-08T00:29:31.271635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T06:19:18.847185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":800,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:07.678212+0000","last_change":"2024-05-08T00:29:27.459396+0000","last_active":"2024-05-08T01:01:07.678212+0000","last_peered":"2024-05-08T01:01:07.678212+0000","last_clean":"2024-05-08T01:01:07.678212+0000","last_became_active":"2024-05-08T00:29:27.458282+0000","last_became_peered":"2024-05-08T00:29:27.458282+0000","last_unstale":"2024-05-08T01:01:07.678212+0000","last_undegraded":"2024-05-08T01:01:07.678212+0000","last_fullsized":"2024-05-08T01:01:07.678212+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T00:29:26.256116+0000","last_clean_scrub_stamp":"2024-05-08T00:29:26.256116+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T05:00:43.917924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738759,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801172,"kb_used_data":2696,"kb_used_omap":118,"kb_used_meta":1798345,"kb_avail":92570668,"statfs":{"total":96636764160,"available":94792364032,"internally_reserved":0,"allocated":2760704,"data_stored":1460396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121245,"internal_metadata":1841505891},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738760,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1421780,"kb_used_data":2756,"kb_used_omap":161,"kb_used_meta":1418846,"kb_avail":92950060,"statfs":{"total":96636764160,"available":95180861440,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":165279,"internal_metadata":1452898913},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738758,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1184020,"kb_used_data":2756,"kb_used_omap":194,"kb_used_meta":1181053,"kb_avail":93187820,"statfs":{"total":96636764160,"available":95424327680,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":198729,"internal_metadata":1209399223},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:01:43.964 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-08T00:29:31.271635+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=29, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:43.964 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-08T00:29:31.271635+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=29, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:43.964 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-08T00:29:31.271635+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=29, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:43.964 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-08T00:29:31.271635+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=29, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:43.964 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-08T00:29:31.271635+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=29, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:43.965 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-08T00:29:31.271635+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=29, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:43.965 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-08T00:29:31.271635+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=29, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:43.965 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-08T00:29:31.271635+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=29, tm_sec=31, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:43.965 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-08T00:29:26.256116+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=0, tm_min=29, tm_sec=26, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=1, tm_min=1, tm_sec=41, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T01:01:43.965 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-08T01:02:03.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-08T01:02:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-05-08T01:02:04.276 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-05-08T01:02:04.288 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1312,"stamp":"2024-05-08T01:02:02.427980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4406952,"kb_used_data":8208,"kb_used_omap":473,"kb_used_meta":4398246,"kb_avail":278708568,"statfs":{"total":289910292480,"available":285397573632,"internally_reserved":0,"allocated":8404992,"data_stored":5561924,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":485253,"internal_metadata":4503804027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002334"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":674,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:47.554115+0000","last_change":"2024-05-08T01:01:47.554136+0000","last_active":"2024-05-08T01:01:47.554115+0000","last_peered":"2024-05-08T01:01:47.554115+0000","last_clean":"2024-05-08T01:01:47.554115+0000","last_became_active":"2024-05-08T00:29:32.298235+0000","last_became_peered":"2024-05-08T00:29:32.298235+0000","last_unstale":"2024-05-08T01:01:47.554115+0000","last_undegraded":"2024-05-08T01:01:47.554115+0000","last_fullsized":"2024-05-08T01:01:47.554115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:47.554008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:47.554008+0000","last_clean_scrub_stamp":"2024-05-08T01:01:47.554008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:37:28.425809+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":675,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:46.433191+0000","last_change":"2024-05-08T01:01:46.433191+0000","last_active":"2024-05-08T01:01:46.433191+0000","last_peered":"2024-05-08T01:01:46.433191+0000","last_clean":"2024-05-08T01:01:46.433191+0000","last_became_active":"2024-05-08T00:29:32.296922+0000","last_became_peered":"2024-05-08T00:29:32.296922+0000","last_unstale":"2024-05-08T01:01:46.433191+0000","last_undegraded":"2024-05-08T01:01:46.433191+0000","last_fullsized":"2024-05-08T01:01:46.433191+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:46.433049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:46.433049+0000","last_clean_scrub_stamp":"2024-05-08T01:01:46.433049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T11:12:29.142985+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":675,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:45.506448+0000","last_change":"2024-05-08T01:01:45.506471+0000","last_active":"2024-05-08T01:01:45.506448+0000","last_peered":"2024-05-08T01:01:45.506448+0000","last_clean":"2024-05-08T01:01:45.506448+0000","last_became_active":"2024-05-08T00:29:32.295815+0000","last_became_peered":"2024-05-08T00:29:32.295815+0000","last_unstale":"2024-05-08T01:01:45.506448+0000","last_undegraded":"2024-05-08T01:01:45.506448+0000","last_fullsized":"2024-05-08T01:01:45.506448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:45.506348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:45.506348+0000","last_clean_scrub_stamp":"2024-05-08T01:01:45.506348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T07:46:53.669461+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":675,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:48.356875+0000","last_change":"2024-05-08T01:01:48.356902+0000","last_active":"2024-05-08T01:01:48.356875+0000","last_peered":"2024-05-08T01:01:48.356875+0000","last_clean":"2024-05-08T01:01:48.356875+0000","last_became_active":"2024-05-08T00:29:32.298232+0000","last_became_peered":"2024-05-08T00:29:32.298232+0000","last_unstale":"2024-05-08T01:01:48.356875+0000","last_undegraded":"2024-05-08T01:01:48.356875+0000","last_fullsized":"2024-05-08T01:01:48.356875+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:48.356773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:48.356773+0000","last_clean_scrub_stamp":"2024-05-08T01:01:48.356773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:15:51.599037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":676,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:42.811995+0000","last_change":"2024-05-08T01:01:42.812017+0000","last_active":"2024-05-08T01:01:42.811995+0000","last_peered":"2024-05-08T01:01:42.811995+0000","last_clean":"2024-05-08T01:01:42.811995+0000","last_became_active":"2024-05-08T00:29:32.297691+0000","last_became_peered":"2024-05-08T00:29:32.297691+0000","last_unstale":"2024-05-08T01:01:42.811995+0000","last_undegraded":"2024-05-08T01:01:42.811995+0000","last_fullsized":"2024-05-08T01:01:42.811995+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:42.811894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:42.811894+0000","last_clean_scrub_stamp":"2024-05-08T01:01:42.811894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:18:28.975052+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":674,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:43.984689+0000","last_change":"2024-05-08T01:01:43.984711+0000","last_active":"2024-05-08T01:01:43.984689+0000","last_peered":"2024-05-08T01:01:43.984689+0000","last_clean":"2024-05-08T01:01:43.984689+0000","last_became_active":"2024-05-08T00:29:32.295343+0000","last_became_peered":"2024-05-08T00:29:32.295343+0000","last_unstale":"2024-05-08T01:01:43.984689+0000","last_undegraded":"2024-05-08T01:01:43.984689+0000","last_fullsized":"2024-05-08T01:01:43.984689+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:43.984598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:43.984598+0000","last_clean_scrub_stamp":"2024-05-08T01:01:43.984598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T10:58:20.369170+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":675,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:44.997065+0000","last_change":"2024-05-08T01:01:44.997097+0000","last_active":"2024-05-08T01:01:44.997065+0000","last_peered":"2024-05-08T01:01:44.997065+0000","last_clean":"2024-05-08T01:01:44.997065+0000","last_became_active":"2024-05-08T00:29:32.297247+0000","last_became_peered":"2024-05-08T00:29:32.297247+0000","last_unstale":"2024-05-08T01:01:44.997065+0000","last_undegraded":"2024-05-08T01:01:44.997065+0000","last_fullsized":"2024-05-08T01:01:44.997065+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:44.996967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:44.996967+0000","last_clean_scrub_stamp":"2024-05-08T01:01:44.996967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T04:03:22.745848+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":675,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:44.535071+0000","last_change":"2024-05-08T01:01:44.535093+0000","last_active":"2024-05-08T01:01:44.535071+0000","last_peered":"2024-05-08T01:01:44.535071+0000","last_clean":"2024-05-08T01:01:44.535071+0000","last_became_active":"2024-05-08T00:29:32.295380+0000","last_became_peered":"2024-05-08T00:29:32.295380+0000","last_unstale":"2024-05-08T01:01:44.535071+0000","last_undegraded":"2024-05-08T01:01:44.535071+0000","last_fullsized":"2024-05-08T01:01:44.535071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T01:01:44.534976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T01:01:44.534976+0000","last_clean_scrub_stamp":"2024-05-08T01:01:44.534976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T02:49:25.232887+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"335'99","reported_seq":811,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-05-08T01:01:43.368611+0000","last_change":"2024-05-08T01:01:43.368641+0000","last_active":"2024-05-08T01:01:43.368611+0000","last_peered":"2024-05-08T01:01:43.368611+0000","last_clean":"2024-05-08T01:01:43.368611+0000","last_became_active":"2024-05-08T00:29:27.458282+0000","last_became_peered":"2024-05-08T00:29:27.458282+0000","last_unstale":"2024-05-08T01:01:43.368611+0000","last_undegraded":"2024-05-08T01:01:43.368611+0000","last_fullsized":"2024-05-08T01:01:43.368611+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'99","last_scrub_stamp":"2024-05-08T01:01:43.368539+0000","last_deep_scrub":"335'99","last_deep_scrub_stamp":"2024-05-08T01:01:43.368539+0000","last_clean_scrub_stamp":"2024-05-08T01:01:43.368539+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T03:07:15.920665+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738763,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801168,"kb_used_data":2696,"kb_used_omap":118,"kb_used_meta":1798345,"kb_avail":92570672,"statfs":{"total":96636764160,"available":94792368128,"internally_reserved":0,"allocated":2760704,"data_stored":1460396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":121245,"internal_metadata":1841505891},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738764,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1421772,"kb_used_data":2756,"kb_used_omap":161,"kb_used_meta":1418846,"kb_avail":92950068,"statfs":{"total":96636764160,"available":95180869632,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":165279,"internal_metadata":1452898913},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738762,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1184012,"kb_used_data":2756,"kb_used_omap":194,"kb_used_meta":1181053,"kb_avail":93187828,"statfs":{"total":96636764160,"available":95424335872,"internally_reserved":0,"allocated":2822144,"data_stored":2050764,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":198729,"internal_metadata":1209399223},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T01:02:04.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-05-08T01:02:04.605 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-08T01:02:04.605 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-08T01:02:04.606 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-05-08T01:02:04.606 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:02:04.689 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-05-08T01:02:04.689 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-05-08T01:02:04.689 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:02:04.755 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-05-08T01:02:04.756 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-05-08T01:02:04.756 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:02:04.839 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-05-08T01:02:04.839 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-08T01:02:04.840 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-05-08T01:02:04.840 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:02:04.865 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-05-08T01:02:04.866 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-08T01:02:04.866 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-05-08T01:02:04.866 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T01:02:04.918 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-05-08T01:02:04.919 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-08T01:02:04.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-05-08T01:02:05.030 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T01:02:05.031 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-05-08T01:02:05.394 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T01:02:05.394 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-05-08T01:02:05.673 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T01:02:05.673 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-05-08T01:02:06.001 INFO:tasks.ceph:Archiving mon data... 2024-05-08T01:02:06.001 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630/data/cluster1.mon.a.tgz 2024-05-08T01:02:06.001 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-05-08T01:02:06.030 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.vO7MMfUal0 2024-05-08T01:02:06.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.vO7MMfUal0 2024-05-08T01:02:06.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.vO7MMfUal0 2024-05-08T01:02:06.310 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.vO7MMfUal0 is 393KB 2024-05-08T01:02:06.406 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.vO7MMfUal0 2024-05-08T01:02:06.423 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-08T01:02:06.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-05-08T01:02:06.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-05-08T01:02:06.545 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-08T01:02:06.563 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-08T01:02:06.563 INFO:tasks.ceph:Archiving crash dumps... 2024-05-08T01:02:06.566 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630/remote/smithi064/crash 2024-05-08T01:02:06.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-08T01:02:06.611 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630/remote/smithi116/crash 2024-05-08T01:02:06.612 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-08T01:02:06.641 INFO:tasks.ceph:Compressing logs... 2024-05-08T01:02:06.641 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-08T01:02:06.656 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-08T01:02:06.679 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31400.log 2024-05-08T01:02:06.679 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34782.log 2024-05-08T01:02:06.693 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34869.log 2024-05-08T01:02:06.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.31400.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31400.log.gz 2024-05-08T01:02:06.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-05-08T01:02:06.693 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34782.log.gz 2024-05-08T01:02:06.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35163.log 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34869.log.gz 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34575.log 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.35163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35015.log 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35163.log.gz 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35053.log 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34575.log.gz 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.35015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36174.log 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35015.log.gz 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.35053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31579.log 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35053.log.gz 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35205.log 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.36174.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36174.log.gz 2024-05-08T01:02:06.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.31579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35121.log 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31579.log.gz 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5/var/log/ceph/cluster1-client.admin.35205.log: --verbose -- /var/log/ceph/cluster1-client.admin.36090.log 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35205.log.gz 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.35121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34909.log 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35121.log.gz 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.36090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34659.log 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36090.log.gz 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34477.log 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34909.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34909.log.gz 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34701.log 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34659.log.gz 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35922.log 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34477.log.gz 2024-05-08T01:02:06.695 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/cluster1-client.admin.34701.log: -5 --verbose -- /var/log/ceph/cluster1.log 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34701.log.gz 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.35922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36048.log 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35922.log.gz 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36384.log 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.36048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34533.log 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36048.log.gz 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.36384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34393.log 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36384.log.gz 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/cluster1-client.admin.34533.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.34617.log 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34533.log.gz 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34393.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34393.log.gz 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34781.log 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36006.log 2024-05-08T01:02:06.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34617.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34617.log.gz 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31480.log 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34781.log.gz 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.36006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36006.log.gz 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.31480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36258.log 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31480.log.gz 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31652.log 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-osd.0.log: /var/log/ceph/cluster1-client.admin.36258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34995.log 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36258.log.gz 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.31652.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31652.log.gz 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36216.log 2024-05-08T01:02:06.697 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34995.log.gz 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36424.log 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.36216.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26775.log 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36216.log.gz 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.36424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36424.log.gz 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.tmp-client.admin.26775.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26775.log.gz 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35964.log 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31440.log 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34783.log 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.35964.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35964.log.gz 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.31440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31242.log 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31440.log.gz 2024-05-08T01:02:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34908.log 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34783.log.gz 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.31242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36132.log 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31242.log.gz 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.34908.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34908.log.gz 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.36132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36342.log 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36132.log.gz 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-client.admin.36342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36300.log 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36342.log.gz 2024-05-08T01:02:06.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cluster1-osd.2.log: /var/log/ceph/cluster1-client.admin.36300.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36300.log.gz 2024-05-08T01:02:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85968.log 2024-05-08T01:02:06.734 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51226.log 2024-05-08T01:02:06.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70268.log 2024-05-08T01:02:06.736 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85968.log.gz 2024-05-08T01:02:06.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90904.log 2024-05-08T01:02:06.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.70268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48574.log 2024-05-08T01:02:06.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74311.log 2024-05-08T01:02:06.736 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90904.log.gz 2024-05-08T01:02:06.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.48574.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93684.log 2024-05-08T01:02:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74311.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51335.log 2024-05-08T01:02:06.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93684.log: gzip 91.6% -5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.70268.log.gz -- 2024-05-08T01:02:06.737 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/cluster2-client.mirror.43326.log 2024-05-08T01:02:06.739 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51335.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85775.log 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75505.log 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93684.log.gz 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65524.log 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55893.log 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75505.log.gz 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.65524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57004.log 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50893.log 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.55893.log: /var/log/ceph/cluster2-client.mirror.57004.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55893.log.gz 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85775.log.gz 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77864.log 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44973.log 2024-05-08T01:02:06.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50893.log: /var/log/ceph/cluster1-client.mirror.77864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89599.log 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84033.log 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77864.log.gz 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50114.log 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89599.log.gz 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.84033.log: /var/log/ceph/cluster1-client.mirror.90139.log 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39887.log 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47769.log 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90139.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.90139.log.gz 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42127.log 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75119.log 2024-05-08T01:02:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53542.log 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42127.log: /var/log/ceph/cluster2-client.mirror.75119.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34890.log 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.53542.log: -- replaced with /var/log/ceph/cluster2-client.mirror.75119.log.gz 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74355.log 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34890.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34890.log.gz 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52136.log 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50270.log 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74355.log.gz 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77145.log 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.48574.log.gz 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91771.log 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.77145.log: /var/log/ceph/cluster1-client.mirror.50270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44260.log 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42592.log 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44260.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77145.log.gz 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51226.log.gz 2024-05-08T01:02:06.752 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88172.log 2024-05-08T01:02:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42592.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44973.log.gz 2024-05-08T01:02:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84285.log 2024-05-08T01:02:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88172.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44260.log.gz 2024-05-08T01:02:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78303.log 2024-05-08T01:02:06.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84285.log: 90.7% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84285.log.gz 2024-05-08T01:02:06.757 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.88172.log.gz 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35562.log 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.35562.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35562.log.gz 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67371.log 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66918.log 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.67371.log: /var/log/ceph/cluster2-client.mirror.78303.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78303.log.gz 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67371.log.gz 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79889.log 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6%/var/log/ceph/cluster2-client.mirror.66918.log: -- replaced with /var/log/ceph/cluster2-client.mirror.43326.log.gz 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66918.log.gz 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92841.log 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91016.log 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79889.log.gz 2024-05-08T01:02:06.758 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:06.763 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92841.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92841.log.gz 2024-05-08T01:02:06.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69512.log 2024-05-08T01:02:06.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68202.log 2024-05-08T01:02:06.763 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3%/var/log/ceph/cluster2-client.mirror.69512.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39887.log.gz/var/log/ceph/cluster1-client.mirror.91016.log: 2024-05-08T01:02:06.763 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91016.log.gz 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69512.log.gz 2024-05-08T01:02:06.763 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:06.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49705.log 2024-05-08T01:02:06.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36024.log 2024-05-08T01:02:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.49705.log: /var/log/ceph/cluster2-client.mirror.68202.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42127.log.gz 2024-05-08T01:02:06.764 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42592.log.gz 2024-05-08T01:02:06.764 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52136.log.gz 2024-05-08T01:02:06.768 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50114.log.gz 2024-05-08T01:02:06.770 INFO:teuthology.orchestra.run.smithi116.stderr: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68202.log.gz 2024-05-08T01:02:06.771 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46445.log 2024-05-08T01:02:06.771 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36024.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36024.log.gz 2024-05-08T01:02:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49319.log 2024-05-08T01:02:06.772 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46445.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68651.log 2024-05-08T01:02:06.773 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91899.log 2024-05-08T01:02:06.774 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.68651.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68651.log.gz 2024-05-08T01:02:06.778 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69862.log 2024-05-08T01:02:06.778 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66246.log 2024-05-08T01:02:06.778 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49705.log.gz 2024-05-08T01:02:06.778 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91899.log: /var/log/ceph/cluster2-client.mirror.69862.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51335.log.gz 2024-05-08T01:02:06.778 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91899.log.gz 2024-05-08T01:02:06.778 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36902.log 2024-05-08T01:02:06.779 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.66246.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69862.log.gz 2024-05-08T01:02:06.779 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66246.log.gz 2024-05-08T01:02:06.780 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94316.log 2024-05-08T01:02:06.780 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.0.36902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39759.log 2024-05-08T01:02:06.783 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.94316.log: /var/log/ceph/cluster1-client.mirror.92137.log 2024-05-08T01:02:06.783 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94316.log.gz 2024-05-08T01:02:06.783 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94184.log 2024-05-08T01:02:06.783 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92137.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92641.log 2024-05-08T01:02:06.783 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92137.log.gz 2024-05-08T01:02:06.783 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39759.log.gz 2024-05-08T01:02:06.787 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54107.log 2024-05-08T01:02:06.788 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94184.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92641.log.gz 2024-05-08T01:02:06.788 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38264.log 2024-05-08T01:02:06.788 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.54107.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54107.log.gz 2024-05-08T01:02:06.788 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82386.log 2024-05-08T01:02:06.788 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49673.log 2024-05-08T01:02:06.789 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.82386.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38264.log.gz 2024-05-08T01:02:06.789 INFO:teuthology.orchestra.run.smithi116.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.82386.log.gz 2024-05-08T01:02:06.791 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70668.log 2024-05-08T01:02:06.791 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49673.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50893.log.gz 2024-05-08T01:02:06.791 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90229.log 2024-05-08T01:02:06.791 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49673.log.gz 2024-05-08T01:02:06.793 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.70668.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46445.log.gz 2024-05-08T01:02:06.796 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50270.log.gz 2024-05-08T01:02:06.799 INFO:teuthology.orchestra.run.smithi116.stderr: 20.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36902.log.gz 2024-05-08T01:02:06.803 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85826.log 2024-05-08T01:02:06.804 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90229.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90229.log.gz 2024-05-08T01:02:06.810 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49391.log 2024-05-08T01:02:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85826.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85826.log.gz 2024-05-08T01:02:06.824 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43233.log 2024-05-08T01:02:06.836 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35856.log 2024-05-08T01:02:06.837 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49391.log.gz 2024-05-08T01:02:06.848 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43233.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43233.log.gz 2024-05-08T01:02:06.849 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89374.log 2024-05-08T01:02:06.849 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65524.log.gz/var/log/ceph/cluster1-client.admin.35856.log: 2024-05-08T01:02:06.849 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35856.log.gz 2024-05-08T01:02:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51895.log 2024-05-08T01:02:06.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89374.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89374.log.gz 2024-05-08T01:02:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38361.log 2024-05-08T01:02:06.853 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37007.log 2024-05-08T01:02:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90971.log 2024-05-08T01:02:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.37007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44167.log 2024-05-08T01:02:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90971.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90971.log.gz 2024-05-08T01:02:06.855 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87401.log 2024-05-08T01:02:06.856 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78380.log 2024-05-08T01:02:06.860 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87401.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38361.log.gz 2024-05-08T01:02:06.863 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44167.log.gz 2024-05-08T01:02:06.866 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87401.log.gz 2024-05-08T01:02:06.866 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91420.log 2024-05-08T01:02:06.871 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.78380.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.78380.log.gz 2024-05-08T01:02:06.871 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89194.log 2024-05-08T01:02:06.872 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91420.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91420.log.gz 2024-05-08T01:02:06.872 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86766.log 2024-05-08T01:02:06.873 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89194.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89194.log.gz 2024-05-08T01:02:06.875 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43291.log 2024-05-08T01:02:06.877 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86766.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86766.log.gz 2024-05-08T01:02:06.877 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41658.log 2024-05-08T01:02:06.877 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51069.log 2024-05-08T01:02:06.878 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44880.log 2024-05-08T01:02:06.880 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.51069.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43291.log.gz 2024-05-08T01:02:06.880 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45380.log 2024-05-08T01:02:06.880 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41658.log.gz 2024-05-08T01:02:06.880 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86388.log 2024-05-08T01:02:06.880 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77446.log 2024-05-08T01:02:06.885 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86388.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86388.log.gz 2024-05-08T01:02:06.885 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92412.log 2024-05-08T01:02:06.885 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77446.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77446.log.gz 2024-05-08T01:02:06.885 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45865.log 2024-05-08T01:02:06.888 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92412.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.92412.log.gz 2024-05-08T01:02:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50559.log 2024-05-08T01:02:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.45865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38512.log 2024-05-08T01:02:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50559.log: 88.3% 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70668.log.gz 2024-05-08T01:02:06.893 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44880.log.gz 2024-05-08T01:02:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38419.log 2024-05-08T01:02:06.893 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45380.log.gz 2024-05-08T01:02:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85897.log 2024-05-08T01:02:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49189.log 2024-05-08T01:02:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38419.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45865.log.gz 2024-05-08T01:02:06.894 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40410.log 2024-05-08T01:02:06.894 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38512.log.gz 2024-05-08T01:02:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85897.log: /var/log/ceph/cluster1-client.mirror.49189.log: gzip 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85897.log.gz 2024-05-08T01:02:06.896 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72175.log 2024-05-08T01:02:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81963.log 2024-05-08T01:02:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.72175.log: /var/log/ceph/cluster2-client.mirror.40410.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38419.log.gz 2024-05-08T01:02:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45531.log 2024-05-08T01:02:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.81963.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81963.log.gz 2024-05-08T01:02:06.903 INFO:teuthology.orchestra.run.smithi116.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72175.log.gz 2024-05-08T01:02:06.903 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67052.log 2024-05-08T01:02:06.903 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40410.log.gz 2024-05-08T01:02:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40786.log 2024-05-08T01:02:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91219.log 2024-05-08T01:02:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40786.log: /var/log/ceph/cluster2-client.mirror.67052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48339.log 2024-05-08T01:02:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91219.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.91219.log.gz 2024-05-08T01:02:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52589.log 2024-05-08T01:02:06.903 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73703.log 2024-05-08T01:02:06.904 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66953.log 2024-05-08T01:02:06.904 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45531.log.gz 2024-05-08T01:02:06.905 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.73703.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40786.log.gz 2024-05-08T01:02:06.905 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48339.log.gz 2024-05-08T01:02:06.906 INFO:teuthology.orchestra.run.smithi116.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73703.log.gz 2024-05-08T01:02:06.907 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87903.log 2024-05-08T01:02:06.914 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.66953.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66953.log.gz 2024-05-08T01:02:06.939 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40724.log 2024-05-08T01:02:06.942 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87903.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87903.log.gz 2024-05-08T01:02:06.965 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52159.log 2024-05-08T01:02:06.967 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40724.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40724.log.gz 2024-05-08T01:02:06.973 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91264.log 2024-05-08T01:02:06.978 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52159.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74311.log.gz 2024-05-08T01:02:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89846.log 2024-05-08T01:02:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91264.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.91264.log.gz 2024-05-08T01:02:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44225.log 2024-05-08T01:02:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89846.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89846.log.gz 2024-05-08T01:02:06.981 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45159.log 2024-05-08T01:02:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89464.log 2024-05-08T01:02:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70720.log 2024-05-08T01:02:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89464.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89464.log.gz 2024-05-08T01:02:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50959.log 2024-05-08T01:02:06.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.70720.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70720.log.gz 2024-05-08T01:02:06.987 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37676.log 2024-05-08T01:02:06.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52869.log 2024-05-08T01:02:06.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92365.log 2024-05-08T01:02:06.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48076.log 2024-05-08T01:02:06.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92365.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44225.log.gz 2024-05-08T01:02:06.989 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45159.log.gz 2024-05-08T01:02:06.993 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.94184.log.gz 2024-05-08T01:02:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49065.log 2024-05-08T01:02:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51026.log 2024-05-08T01:02:06.996 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51201.log 2024-05-08T01:02:06.997 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.51026.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49065.log.gz 2024-05-08T01:02:06.997 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84761.log 2024-05-08T01:02:06.997 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84494.log 2024-05-08T01:02:06.999 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84761.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84761.log.gz 2024-05-08T01:02:06.999 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71133.log 2024-05-08T01:02:07.000 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84494.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84494.log.gz 2024-05-08T01:02:07.000 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71054.log 2024-05-08T01:02:07.000 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.71133.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71133.log.gz 2024-05-08T01:02:07.001 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35646.log 2024-05-08T01:02:07.002 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.71054.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51201.log.gz 2024-05-08T01:02:07.010 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45798.log 2024-05-08T01:02:07.011 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.35646.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35646.log.gz 2024-05-08T01:02:07.015 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67052.log.gz 2024-05-08T01:02:07.018 INFO:teuthology.orchestra.run.smithi116.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37007.log.gz 2024-05-08T01:02:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49211.log 2024-05-08T01:02:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92464.log 2024-05-08T01:02:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88492.log 2024-05-08T01:02:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.92464.log: /var/log/ceph/cluster2-client.mirror.45473.log 2024-05-08T01:02:07.027 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.88492.log: -- replaced with /var/log/ceph/cluster2-client.mirror.45798.log.gz 2024-05-08T01:02:07.028 INFO:teuthology.orchestra.run.smithi116.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92464.log.gz 2024-05-08T01:02:07.028 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88492.log.gz 2024-05-08T01:02:07.036 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45628.log 2024-05-08T01:02:07.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51159.log 2024-05-08T01:02:07.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45628.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45473.log.gz 2024-05-08T01:02:07.039 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45628.log.gz 2024-05-08T01:02:07.040 INFO:teuthology.orchestra.run.smithi116.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49189.log.gz 2024-05-08T01:02:07.043 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48076.log.gz 2024-05-08T01:02:07.046 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92277.log 2024-05-08T01:02:07.047 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35688.log 2024-05-08T01:02:07.051 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92277.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92277.log.gz 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51159.log.gz 2024-05-08T01:02:07.051 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:07.051 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86039.log 2024-05-08T01:02:07.051 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35688.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35688.log.gz 2024-05-08T01:02:07.051 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93981.log 2024-05-08T01:02:07.054 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86039.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86039.log.gz 2024-05-08T01:02:07.054 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49956.log 2024-05-08T01:02:07.054 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50446.log 2024-05-08T01:02:07.054 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93981.log: /var/log/ceph/cluster2-client.mirror.49956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69882.log 2024-05-08T01:02:07.054 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94575.log 2024-05-08T01:02:07.056 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77033.log 2024-05-08T01:02:07.056 INFO:teuthology.orchestra.run.smithi116.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93981.log.gz 2024-05-08T01:02:07.056 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.94575.log: /var/log/ceph/cluster1-client.mirror.69882.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49956.log.gz 2024-05-08T01:02:07.056 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94575.log.gz 2024-05-08T01:02:07.057 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51026.log.gz 2024-05-08T01:02:07.057 INFO:teuthology.orchestra.run.smithi116.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.69882.log.gz 2024-05-08T01:02:07.063 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69991.log 2024-05-08T01:02:07.064 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77033.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77033.log.gz 2024-05-08T01:02:07.064 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66985.log 2024-05-08T01:02:07.072 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69991.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69991.log.gz 2024-05-08T01:02:07.072 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45066.log 2024-05-08T01:02:07.072 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.66985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93223.log 2024-05-08T01:02:07.072 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66985.log.gz 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45066.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45066.log.gz 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82792.log 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51069.log.gz 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36563.log 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82792.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93223.log.gz 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82792.log.gz 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78175.log 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39604.log 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.78175.log: /var/log/ceph/cluster1-client.mirror.36563.log: gzip 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78175.log.gz 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92494.log 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93349.log 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39604.log: /var/log/ceph/cluster1-client.mirror.92494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39080.log 2024-05-08T01:02:07.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93349.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.92494.log.gz 2024-05-08T01:02:07.074 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93349.log.gz 2024-05-08T01:02:07.074 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86088.log 2024-05-08T01:02:07.074 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.39080.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39604.log.gz 2024-05-08T01:02:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87381.log 2024-05-08T01:02:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86088.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86088.log.gz 2024-05-08T01:02:07.076 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80838.log 2024-05-08T01:02:07.077 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87381.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87381.log.gz 2024-05-08T01:02:07.077 INFO:teuthology.orchestra.run.smithi116.stderr: 63.7% -- replaced with /var/log/ceph/cluster1-client.mirror.36563.log.gz 2024-05-08T01:02:07.079 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82408.log 2024-05-08T01:02:07.079 INFO:teuthology.orchestra.run.smithi116.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80838.log.gz 2024-05-08T01:02:07.079 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36819.log 2024-05-08T01:02:07.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82408.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50959.log.gz 2024-05-08T01:02:07.090 INFO:teuthology.orchestra.run.smithi116.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53542.log.gz 2024-05-08T01:02:07.091 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75995.log 2024-05-08T01:02:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.36819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74647.log 2024-05-08T01:02:07.093 INFO:teuthology.orchestra.run.smithi116.stderr: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36819.log.gz 2024-05-08T01:02:07.093 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.75995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50669.log 2024-05-08T01:02:07.095 INFO:teuthology.orchestra.run.smithi116.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75995.log.gz 2024-05-08T01:02:07.095 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92667.log 2024-05-08T01:02:07.096 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76361.log 2024-05-08T01:02:07.101 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92667.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74647.log.gz 2024-05-08T01:02:07.102 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88843.log 2024-05-08T01:02:07.104 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.76361.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57004.log.gz 2024-05-08T01:02:07.106 INFO:teuthology.orchestra.run.smithi116.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76361.log.gz 2024-05-08T01:02:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41065.log 2024-05-08T01:02:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.37007.log 2024-05-08T01:02:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41065.log: /var/log/ceph/cluster2-client.mirror.88843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48860.log 2024-05-08T01:02:07.115 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88843.log.gz 2024-05-08T01:02:07.116 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62896.log 2024-05-08T01:02:07.116 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75453.log 2024-05-08T01:02:07.116 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.62896.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62896.log.gz 2024-05-08T01:02:07.117 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.0.37007.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41065.log.gz 2024-05-08T01:02:07.117 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87452.log 2024-05-08T01:02:07.119 INFO:teuthology.orchestra.run.smithi064.stderr: 87.5% -- replaced with /var/log/ceph/cluster1.log.gz 2024-05-08T01:02:07.121 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75453.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48860.log.gz 2024-05-08T01:02:07.121 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77125.log 2024-05-08T01:02:07.121 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87452.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87452.log.gz 2024-05-08T01:02:07.121 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46852.log 2024-05-08T01:02:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77125.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77125.log.gz 2024-05-08T01:02:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50473.log 2024-05-08T01:02:07.124 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85548.log 2024-05-08T01:02:07.126 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38326.log 2024-05-08T01:02:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85548.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46852.log.gz 2024-05-08T01:02:07.129 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85548.log.gz 2024-05-08T01:02:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89936.log 2024-05-08T01:02:07.129 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89104.log 2024-05-08T01:02:07.131 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50559.log.gz 2024-05-08T01:02:07.131 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38326.log.gz 2024-05-08T01:02:07.131 INFO:teuthology.orchestra.run.smithi064.stderr: 94.2% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-05-08T01:02:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89936.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89936.log.gz 2024-05-08T01:02:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91527.log 2024-05-08T01:02:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89104.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89104.log.gz 2024-05-08T01:02:07.133 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34764.log 2024-05-08T01:02:07.134 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91527.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75453.log.gz 2024-05-08T01:02:07.135 INFO:teuthology.orchestra.run.smithi116.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91527.log.gz 2024-05-08T01:02:07.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34764.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34764.log.gz 2024-05-08T01:02:07.139 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70369.log 2024-05-08T01:02:07.139 INFO:teuthology.orchestra.run.smithi116.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49211.log.gz 2024-05-08T01:02:07.139 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85917.log 2024-05-08T01:02:07.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.70369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48953.log 2024-05-08T01:02:07.139 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70369.log.gz 2024-05-08T01:02:07.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50979.log 2024-05-08T01:02:07.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42978.log 2024-05-08T01:02:07.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67682.log 2024-05-08T01:02:07.143 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.42978.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85917.log.gz 2024-05-08T01:02:07.143 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48953.log.gz 2024-05-08T01:02:07.143 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48674.log 2024-05-08T01:02:07.143 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.67682.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67682.log.gz 2024-05-08T01:02:07.143 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/cluster2-client.mirror.93182.log 2024-05-08T01:02:07.143 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50037.log 2024-05-08T01:02:07.143 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93182.log: /var/log/ceph/cluster2-client.mirror.48674.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84996.log 2024-05-08T01:02:07.143 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.93182.log.gz 2024-05-08T01:02:07.144 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50037.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48674.log.gz 2024-05-08T01:02:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84996.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84996.log.gz 2024-05-08T01:02:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43481.log 2024-05-08T01:02:07.148 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50037.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51289.log 2024-05-08T01:02:07.148 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34972.log 2024-05-08T01:02:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51289.log: /var/log/ceph/cluster2-client.mirror.43481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93632.log 2024-05-08T01:02:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34972.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34972.log.gz 2024-05-08T01:02:07.148 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52723.log 2024-05-08T01:02:07.151 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93632.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93632.log.gz 2024-05-08T01:02:07.151 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43481.log.gz 2024-05-08T01:02:07.151 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86017.log 2024-05-08T01:02:07.151 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60512.log 2024-05-08T01:02:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86017.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86017.log.gz 2024-05-08T01:02:07.155 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82408.log.gz 2024-05-08T01:02:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.94869.log 2024-05-08T01:02:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90431.log 2024-05-08T01:02:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.60512.log: /var/log/ceph/cluster2-client.admin.94869.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.94869.log.gz 2024-05-08T01:02:07.155 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/cluster1-client.mirror.94034.log 2024-05-08T01:02:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68103.log 2024-05-08T01:02:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90431.log: /var/log/ceph/cluster1-client.mirror.94034.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.94034.log.gz 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90431.log.gz 2024-05-08T01:02:07.156 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81167.log 2024-05-08T01:02:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68750.log 2024-05-08T01:02:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81167.log: /var/log/ceph/cluster2-client.mirror.68103.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81167.log.gz 2024-05-08T01:02:07.156 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68103.log.gz 2024-05-08T01:02:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31677.log 2024-05-08T01:02:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50516.log 2024-05-08T01:02:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.31677.log: /var/log/ceph/cluster2-client.mirror.68750.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31677.log.gz 2024-05-08T01:02:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64117.log 2024-05-08T01:02:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38770.log 2024-05-08T01:02:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.64117.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64117.log.gz 2024-05-08T01:02:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52846.log 2024-05-08T01:02:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91061.log 2024-05-08T01:02:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94429.log 2024-05-08T01:02:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91061.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91061.log.gz 2024-05-08T01:02:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89531.log 2024-05-08T01:02:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.94429.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94429.log.gz 2024-05-08T01:02:07.163 INFO:teuthology.orchestra.run.smithi116.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38770.log.gz 2024-05-08T01:02:07.164 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53299.log 2024-05-08T01:02:07.164 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89531.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89531.log.gz 2024-05-08T01:02:07.165 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68708.log 2024-05-08T01:02:07.165 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53299.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51289.log.gz 2024-05-08T01:02:07.168 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50446.log.gz 2024-05-08T01:02:07.178 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50669.log.gz 2024-05-08T01:02:07.178 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41716.log 2024-05-08T01:02:07.180 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.68708.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68708.log.gz 2024-05-08T01:02:07.190 INFO:teuthology.orchestra.run.smithi116.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52846.log.gz 2024-05-08T01:02:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49169.log 2024-05-08T01:02:07.196 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41716.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41716.log.gz 2024-05-08T01:02:07.197 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68068.log 2024-05-08T01:02:07.199 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49169.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84033.log.gz 2024-05-08T01:02:07.199 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50149.log 2024-05-08T01:02:07.199 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.68068.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68068.log.gz 2024-05-08T01:02:07.201 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48304.log 2024-05-08T01:02:07.202 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49142.log 2024-05-08T01:02:07.203 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80094.log 2024-05-08T01:02:07.205 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50149.log.gz 2024-05-08T01:02:07.205 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48304.log.gz 2024-05-08T01:02:07.207 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50516.log.gz 2024-05-08T01:02:07.208 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.49142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82366.log 2024-05-08T01:02:07.209 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.80094.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68750.log.gz 2024-05-08T01:02:07.209 INFO:teuthology.orchestra.run.smithi116.stderr: 91.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67717.log 2024-05-08T01:02:07.210 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.80094.log.gz 2024-05-08T01:02:07.213 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82366.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82366.log.gz 2024-05-08T01:02:07.213 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54531.log 2024-05-08T01:02:07.213 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46166.log 2024-05-08T01:02:07.213 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.54531.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54531.log.gz 2024-05-08T01:02:07.215 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82824.log 2024-05-08T01:02:07.215 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90791.log 2024-05-08T01:02:07.217 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.82824.log: /var/log/ceph/cluster2-client.mirror.67717.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82824.log.gz 2024-05-08T01:02:07.217 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67717.log.gz 2024-05-08T01:02:07.217 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51398.log 2024-05-08T01:02:07.218 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61503.log 2024-05-08T01:02:07.218 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51398.log: /var/log/ceph/cluster1-client.mirror.90791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37730.log 2024-05-08T01:02:07.218 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90791.log.gz 2024-05-08T01:02:07.220 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.61503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71390.log 84.5% 2024-05-08T01:02:07.220 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61503.log.gz 2024-05-08T01:02:07.220 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87861.log 2024-05-08T01:02:07.221 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.37730.log: /var/log/ceph/cluster2-client.mirror.71390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40224.log 2024-05-08T01:02:07.221 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87861.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87861.log.gz 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89039.log 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40224.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71390.log.gz 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49797.log 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89039.log: gzip 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89039.log.gz 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65576.log 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83613.log 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.65576.log: /var/log/ceph/cluster1-client.mirror.49797.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94529.log 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65576.log.gz 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93117.log 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83613.log: /var/log/ceph/cluster2-client.mirror.94529.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.94529.log.gz 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47472.log 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93117.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93117.log.gz 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.94953.log 2024-05-08T01:02:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73658.log 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.83613.log.gz/var/log/ceph/cluster2-client.admin.94953.log: 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.94953.log.gz 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69037.log 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45663.log 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69037.log: /var/log/ceph/cluster1-client.mirror.73658.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73658.log.gz 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50473.log.gz 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40224.log.gz 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49142.log.gz 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69037.log.gz 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46035.log 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.45663.log: /var/log/ceph/cluster2-client.mirror.87168.log 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45008.log 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87168.log: /var/log/ceph/cluster1-client.admin.46035.log: gzip 88.2% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45890.log 2024-05-08T01:02:07.234 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.87168.log.gz 2024-05-08T01:02:07.235 INFO:teuthology.orchestra.run.smithi064.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-05-08T01:02:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68616.log 2024-05-08T01:02:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45008.log: /var/log/ceph/cluster2-client.mirror.45890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73626.log 2024-05-08T01:02:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71818.log 2024-05-08T01:02:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.68616.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.37730.log.gz 2024-05-08T01:02:07.239 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68616.log.gz 2024-05-08T01:02:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.73626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35982.log 2024-05-08T01:02:07.239 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46166.log.gz 2024-05-08T01:02:07.240 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.71818.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45663.log.gz 2024-05-08T01:02:07.240 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73626.log.gz 2024-05-08T01:02:07.240 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51729.log 2024-05-08T01:02:07.240 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.35982.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35982.log.gz 2024-05-08T01:02:07.240 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66575.log 2024-05-08T01:02:07.242 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45008.log.gz 2024-05-08T01:02:07.242 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.46035.log.gz 2024-05-08T01:02:07.244 INFO:teuthology.orchestra.run.smithi116.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51729.log.gz 2024-05-08T01:02:07.245 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39569.log 2024-05-08T01:02:07.247 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.66575.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66575.log.gz 2024-05-08T01:02:07.247 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38078.log 2024-05-08T01:02:07.248 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54130.log 2024-05-08T01:02:07.249 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38078.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39569.log.gz 2024-05-08T01:02:07.249 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38078.log.gz 2024-05-08T01:02:07.250 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75053.log 2024-05-08T01:02:07.251 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.54130.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50979.log.gz 2024-05-08T01:02:07.256 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91771.log.gz 2024-05-08T01:02:07.257 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84874.log 2024-05-08T01:02:07.257 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39080.log.gz 2024-05-08T01:02:07.259 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.75053.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75053.log.gz 2024-05-08T01:02:07.259 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42464.log 2024-05-08T01:02:07.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84874.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84874.log.gz 2024-05-08T01:02:07.261 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91723.log 2024-05-08T01:02:07.262 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47472.log.gz 2024-05-08T01:02:07.262 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69956.log 2024-05-08T01:02:07.263 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91723.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42464.log.gz 2024-05-08T01:02:07.264 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.91723.log.gz 2024-05-08T01:02:07.264 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91084.log 2024-05-08T01:02:07.267 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69956.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.42978.log.gz 2024-05-08T01:02:07.267 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69956.log.gz 2024-05-08T01:02:07.267 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92862.log 2024-05-08T01:02:07.267 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91084.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91084.log.gz 2024-05-08T01:02:07.267 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44353.log 2024-05-08T01:02:07.269 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92862.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92862.log.gz 2024-05-08T01:02:07.269 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91844.log 2024-05-08T01:02:07.269 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79924.log 2024-05-08T01:02:07.271 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91844.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44353.log.gz 2024-05-08T01:02:07.271 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.91844.log.gz 2024-05-08T01:02:07.271 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92257.log 2024-05-08T01:02:07.272 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79924.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79924.log.gz 2024-05-08T01:02:07.272 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92072.log 2024-05-08T01:02:07.274 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92257.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92257.log.gz 2024-05-08T01:02:07.274 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73591.log 2024-05-08T01:02:07.277 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92072.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49797.log.gz 2024-05-08T01:02:07.278 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51398.log.gz 2024-05-08T01:02:07.286 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45890.log.gz 2024-05-08T01:02:07.287 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61104.log 2024-05-08T01:02:07.287 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.73591.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73591.log.gz 2024-05-08T01:02:07.293 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58363.log 2024-05-08T01:02:07.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.61104.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49169.log.gz 2024-05-08T01:02:07.306 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41158.log 2024-05-08T01:02:07.313 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.58363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93929.log 2024-05-08T01:02:07.316 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41158.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71054.log.gz 2024-05-08T01:02:07.323 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42823.log 2024-05-08T01:02:07.324 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93929.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93929.log.gz 2024-05-08T01:02:07.329 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41158.log.gz 2024-05-08T01:02:07.339 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87501.log 2024-05-08T01:02:07.346 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38968.log 2024-05-08T01:02:07.353 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85804.log 2024-05-08T01:02:07.354 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38968.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87501.log.gz 2024-05-08T01:02:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52358.log 2024-05-08T01:02:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85804.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38968.log.gz 2024-05-08T01:02:07.380 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43946.log 2024-05-08T01:02:07.380 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.85804.log.gz 2024-05-08T01:02:07.384 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52358.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92365.log.gz 2024-05-08T01:02:07.406 INFO:teuthology.orchestra.run.smithi116.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52358.log.gz 2024-05-08T01:02:07.409 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60489.log 2024-05-08T01:02:07.418 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89869.log 2024-05-08T01:02:07.418 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.60489.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60489.log.gz 2024-05-08T01:02:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44915.log 2024-05-08T01:02:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89869.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89869.log.gz 2024-05-08T01:02:07.431 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43946.log.gz 2024-05-08T01:02:07.437 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42823.log.gz 2024-05-08T01:02:07.444 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35772.log 2024-05-08T01:02:07.446 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44915.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44915.log.gz 2024-05-08T01:02:07.453 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78746.log 2024-05-08T01:02:07.453 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35772.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35772.log.gz 2024-05-08T01:02:07.463 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84974.log 2024-05-08T01:02:07.478 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.78746.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78746.log.gz 2024-05-08T01:02:07.479 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50712.log 2024-05-08T01:02:07.481 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84974.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84974.log.gz 2024-05-08T01:02:07.482 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71818.log.gz 2024-05-08T01:02:07.483 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83135.log 2024-05-08T01:02:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46094.log 2024-05-08T01:02:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83135.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83135.log.gz 2024-05-08T01:02:07.488 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40352.log 2024-05-08T01:02:07.488 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94245.log 2024-05-08T01:02:07.488 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84783.log 2024-05-08T01:02:07.488 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.94245.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94245.log.gz 2024-05-08T01:02:07.488 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48895.log 2024-05-08T01:02:07.490 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84783.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84783.log.gz 2024-05-08T01:02:07.490 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86337.log 2024-05-08T01:02:07.491 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56967.log 2024-05-08T01:02:07.491 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86337.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48895.log.gz 2024-05-08T01:02:07.491 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40352.log.gz 2024-05-08T01:02:07.497 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86337.log.gz 2024-05-08T01:02:07.497 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47286.log 2024-05-08T01:02:07.497 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.56967.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56967.log.gz 2024-05-08T01:02:07.497 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47193.log 2024-05-08T01:02:07.497 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39476.log 2024-05-08T01:02:07.497 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88685.log 2024-05-08T01:02:07.498 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77931.log 2024-05-08T01:02:07.498 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88685.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39476.log.gz 2024-05-08T01:02:07.498 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88685.log.gz 2024-05-08T01:02:07.499 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78689.log 2024-05-08T01:02:07.501 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77931.log: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77931.log.gz 2024-05-08T01:02:07.502 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50493.log 2024-05-08T01:02:07.502 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.78689.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78689.log.gz 2024-05-08T01:02:07.504 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85477.log 2024-05-08T01:02:07.504 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47193.log.gz 2024-05-08T01:02:07.504 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50493.log: gzip 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47286.log.gz 2024-05-08T01:02:07.504 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.49291.log 2024-05-08T01:02:07.505 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85477.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85477.log.gz 2024-05-08T01:02:07.507 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75075.log 2024-05-08T01:02:07.507 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.49291.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88223.log 2024-05-08T01:02:07.517 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94339.log 2024-05-08T01:02:07.517 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88223.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88223.log.gz 2024-05-08T01:02:07.530 INFO:teuthology.orchestra.run.smithi116.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53299.log.gz 2024-05-08T01:02:07.532 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88792.log 2024-05-08T01:02:07.532 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.94339.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94339.log.gz 2024-05-08T01:02:07.532 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48371.log 2024-05-08T01:02:07.534 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88792.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88792.log.gz 2024-05-08T01:02:07.534 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67010.log 2024-05-08T01:02:07.535 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.48371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54340.log 2024-05-08T01:02:07.536 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.67010.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48371.log.gz 2024-05-08T01:02:07.538 INFO:teuthology.orchestra.run.smithi116.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67010.log.gz 2024-05-08T01:02:07.538 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82869.log 2024-05-08T01:02:07.550 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.54340.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58363.log.gz 2024-05-08T01:02:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77503.log 2024-05-08T01:02:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.82869.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82869.log.gz 2024-05-08T01:02:07.561 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92538.log 2024-05-08T01:02:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77503.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77503.log.gz 2024-05-08T01:02:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86437.log 2024-05-08T01:02:07.564 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92538.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92538.log.gz 2024-05-08T01:02:07.565 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92517.log 2024-05-08T01:02:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86437.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86437.log.gz 2024-05-08T01:02:07.568 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47003.log 2024-05-08T01:02:07.569 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92517.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.92517.log.gz 2024-05-08T01:02:07.571 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67030.log 2024-05-08T01:02:07.573 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84903.log 2024-05-08T01:02:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.67030.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47003.log.gz 2024-05-08T01:02:07.575 INFO:teuthology.orchestra.run.smithi116.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67030.log.gz 2024-05-08T01:02:07.575 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72197.log 2024-05-08T01:02:07.577 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84903.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84903.log.gz 2024-05-08T01:02:07.577 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58340.log 2024-05-08T01:02:07.587 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72197.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60512.log.gz 2024-05-08T01:02:07.587 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53731.log 2024-05-08T01:02:07.588 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.58340.log: 84.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58340.log.gz 2024-05-08T01:02:07.588 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45566.log 2024-05-08T01:02:07.599 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39003.log 2024-05-08T01:02:07.607 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45566.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45566.log.gz 2024-05-08T01:02:07.610 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37554.log 2024-05-08T01:02:07.619 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39003.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39003.log.gz 2024-05-08T01:02:07.622 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86550.log 2024-05-08T01:02:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37554.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37554.log.gz 2024-05-08T01:02:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78721.log 2024-05-08T01:02:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86550.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86550.log.gz 2024-05-08T01:02:07.657 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92072.log.gz 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52381.log 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92667.log.gz 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.78721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46573.log 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.52381.log: -- replaced with /var/log/ceph/cluster1-client.mirror.78721.log.gz 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92820.log 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41751.log 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.49291.log.gz 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46573.log: /var/log/ceph/cluster1-client.mirror.92820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39662.log 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41751.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.92820.log.gz 2024-05-08T01:02:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46573.log.gz 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.41437.log 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41751.log.gz 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88934.log 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93318.log 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89891.log 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93318.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39662.log.gz 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93318.log.gz 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53929.log 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89891.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89891.log.gz 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58610.log 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53929.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88934.log.gz 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71751.log 2024-05-08T01:02:07.668 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41437.log.gz 2024-05-08T01:02:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.58610.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58610.log.gz 2024-05-08T01:02:07.671 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92579.log 2024-05-08T01:02:07.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.71751.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71751.log.gz 2024-05-08T01:02:07.671 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43609.log 2024-05-08T01:02:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92579.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50712.log.gz 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92579.log.gz 2024-05-08T01:02:07.672 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-05-08T01:02:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88985.log 2024-05-08T01:02:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61104.log.gz 2024-05-08T01:02:07.674 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.59891.log 2024-05-08T01:02:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88985.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88985.log.gz 2024-05-08T01:02:07.676 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43609.log.gz 2024-05-08T01:02:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77068.log 2024-05-08T01:02:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.59891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90409.log 2024-05-08T01:02:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77068.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77068.log.gz 2024-05-08T01:02:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82849.log 2024-05-08T01:02:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90409.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90409.log.gz 2024-05-08T01:02:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93737.log 2024-05-08T01:02:07.684 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82849.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82849.log.gz 2024-05-08T01:02:07.685 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43853.log 2024-05-08T01:02:07.685 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93737.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.93737.log.gz 2024-05-08T01:02:07.686 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84945.log 2024-05-08T01:02:07.686 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50916.log 2024-05-08T01:02:07.690 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84945.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43853.log.gz 2024-05-08T01:02:07.690 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84945.log.gz 2024-05-08T01:02:07.690 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80480.log 2024-05-08T01:02:07.691 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89261.log 2024-05-08T01:02:07.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.80480.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80480.log.gz 2024-05-08T01:02:07.695 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39794.log 2024-05-08T01:02:07.696 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89261.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89261.log.gz 2024-05-08T01:02:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49863.log 2024-05-08T01:02:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39794.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75075.log.gz 2024-05-08T01:02:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37519.log 2024-05-08T01:02:07.700 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58144.log 2024-05-08T01:02:07.700 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49863.log: /var/log/ceph/cluster2-client.mirror.37519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91174.log 2024-05-08T01:02:07.700 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.58144.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87472.log 2024-05-08T01:02:07.700 INFO:teuthology.orchestra.run.smithi116.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58144.log.gz 2024-05-08T01:02:07.700 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41379.log 2024-05-08T01:02:07.700 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87472.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91174.log.gz 2024-05-08T01:02:07.702 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87472.log.gz 2024-05-08T01:02:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94554.log 2024-05-08T01:02:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51445.log 2024-05-08T01:02:07.711 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94554.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37519.log.gz 2024-05-08T01:02:07.711 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.94554.log.gz 2024-05-08T01:02:07.711 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36877.log 2024-05-08T01:02:07.711 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39794.log.gz 2024-05-08T01:02:07.711 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51445.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54317.log 2024-05-08T01:02:07.711 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36877.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36877.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52267.log 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79160.log/var/log/ceph/cluster2-client.mirror.54317.log: 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54317.log.gz 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38454.log 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94055.log 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79160.log.gz 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49863.log.gz 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94055.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94055.log.gz 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41379.log.gz 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91309.log 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87648.log 2024-05-08T01:02:07.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50293.log 2024-05-08T01:02:07.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87648.log: 89.0% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38454.log.gz 2024-05-08T01:02:07.715 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.51445.log.gz 2024-05-08T01:02:07.715 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87648.log.gz 2024-05-08T01:02:07.715 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91309.log.gz 2024-05-08T01:02:07.716 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56529.log 2024-05-08T01:02:07.716 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88314.log 2024-05-08T01:02:07.716 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84377.log 2024-05-08T01:02:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88314.log: /var/log/ceph/cluster2-client.mirror.56529.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88314.log.gz 2024-05-08T01:02:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-05-08T01:02:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84377.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84377.log.gz 2024-05-08T01:02:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88472.log 2024-05-08T01:02:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74741.log 2024-05-08T01:02:07.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88472.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88472.log.gz 2024-05-08T01:02:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64924.log 2024-05-08T01:02:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74741.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74741.log.gz 2024-05-08T01:02:07.735 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46259.log 2024-05-08T01:02:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.64924.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64924.log.gz 2024-05-08T01:02:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83170.log 2024-05-08T01:02:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46259.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88663.log 2024-05-08T01:02:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83170.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83170.log.gz 2024-05-08T01:02:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54130.log.gz 2024-05-08T01:02:07.741 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77167.log 2024-05-08T01:02:07.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46352.log 2024-05-08T01:02:07.743 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77167.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50916.log.gz 2024-05-08T01:02:07.743 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88663.log.gz 2024-05-08T01:02:07.747 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46259.log.gz 2024-05-08T01:02:07.747 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51636.log 2024-05-08T01:02:07.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92192.log 2024-05-08T01:02:07.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.51636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93714.log 2024-05-08T01:02:07.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92192.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.92192.log.gz 2024-05-08T01:02:07.747 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51247.log 2024-05-08T01:02:07.752 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46352.log.gz 2024-05-08T01:02:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93714.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93714.log.gz 2024-05-08T01:02:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50493.log.gz 2024-05-08T01:02:07.752 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77411.log 2024-05-08T01:02:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38724.log 2024-05-08T01:02:07.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77411.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77411.log.gz 2024-05-08T01:02:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.37068.log 2024-05-08T01:02:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36546.log 2024-05-08T01:02:07.754 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.1.37068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86879.log 2024-05-08T01:02:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.36546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80880.log 2024-05-08T01:02:07.763 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86879.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86879.log.gz 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.36546.log.gz 2024-05-08T01:02:07.763 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:07.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88914.log 2024-05-08T01:02:07.763 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80880.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51636.log.gz 2024-05-08T01:02:07.764 INFO:teuthology.orchestra.run.smithi116.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77167.log.gz 2024-05-08T01:02:07.766 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42908.log 2024-05-08T01:02:07.766 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88914.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88914.log.gz 2024-05-08T01:02:07.767 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47600.log 2024-05-08T01:02:07.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42908.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41007.log 2024-05-08T01:02:07.770 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47600.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47600.log.gz 2024-05-08T01:02:07.771 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85435.log 2024-05-08T01:02:07.771 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88705.log 2024-05-08T01:02:07.772 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85435.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85435.log.gz 2024-05-08T01:02:07.773 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57320.log 2024-05-08T01:02:07.774 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88705.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41007.log.gz 2024-05-08T01:02:07.776 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88705.log.gz 2024-05-08T01:02:07.776 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43981.log 2024-05-08T01:02:07.801 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.57320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93087.log 2024-05-08T01:02:07.811 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81644.log 2024-05-08T01:02:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93087.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93087.log.gz 2024-05-08T01:02:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80116.log 2024-05-08T01:02:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81644.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81644.log.gz 2024-05-08T01:02:07.826 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54340.log.gz 2024-05-08T01:02:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39697.log 2024-05-08T01:02:07.839 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80116.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38724.log.gz 2024-05-08T01:02:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44636.log 2024-05-08T01:02:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39697.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50293.log.gz 2024-05-08T01:02:07.842 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39697.log.gz 2024-05-08T01:02:07.848 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51247.log.gz 2024-05-08T01:02:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50072.log 2024-05-08T01:02:07.850 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90836.log 2024-05-08T01:02:07.852 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50072.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44636.log.gz 2024-05-08T01:02:07.857 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50072.log.gz 2024-05-08T01:02:07.862 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80746.log 2024-05-08T01:02:07.862 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.72197.log.gz 2024-05-08T01:02:07.862 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86459.log 2024-05-08T01:02:07.862 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80746.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90836.log.gz 2024-05-08T01:02:07.862 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.80746.log.gz 2024-05-08T01:02:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86408.log 2024-05-08T01:02:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86459.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86459.log.gz 2024-05-08T01:02:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75186.log 2024-05-08T01:02:07.872 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86408.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86408.log.gz 2024-05-08T01:02:07.873 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87761.log 2024-05-08T01:02:07.877 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.75186.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75186.log.gz 2024-05-08T01:02:07.877 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-05-08T01:02:07.881 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87761.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87761.log.gz 2024-05-08T01:02:07.883 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41286.log 2024-05-08T01:02:07.883 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56055.log 2024-05-08T01:02:07.887 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41286.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41286.log.gz 2024-05-08T01:02:07.887 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87217.log 2024-05-08T01:02:07.891 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.56055.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56055.log.gz 2024-05-08T01:02:07.891 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94479.log 2024-05-08T01:02:07.891 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87217.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87217.log.gz 2024-05-08T01:02:07.896 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75839.log 2024-05-08T01:02:07.896 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.94479.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94479.log.gz 2024-05-08T01:02:07.896 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59474.log 2024-05-08T01:02:07.905 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88863.log 2024-05-08T01:02:07.905 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.59474.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59474.log.gz 2024-05-08T01:02:07.906 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80880.log.gz 2024-05-08T01:02:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88821.log 2024-05-08T01:02:07.909 INFO:teuthology.orchestra.run.smithi116.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75839.log.gz/var/log/ceph/cluster2-client.mirror.88863.log: 2024-05-08T01:02:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78360.log 2024-05-08T01:02:07.910 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88821.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88821.log.gz 2024-05-08T01:02:07.910 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75572.log 2024-05-08T01:02:07.912 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.78360.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78360.log.gz 2024-05-08T01:02:07.913 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose 90.6% -- /var/log/ceph/cluster1-client.mirror.90116.log 2024-05-08T01:02:07.913 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.88863.log.gz 2024-05-08T01:02:07.913 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.75572.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75572.log.gz 2024-05-08T01:02:07.916 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47940.log 2024-05-08T01:02:07.916 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90116.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90116.log.gz 2024-05-08T01:02:07.916 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47565.log 2024-05-08T01:02:07.916 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84712.log 2024-05-08T01:02:07.916 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62421.log 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84712.log: 88.5% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84712.log.gz 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47940.log.gz 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85946.log 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.62421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39511.log 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62421.log.gz 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51268.log 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39511.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85946.log.gz 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89081.log 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42685.log 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84352.log 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49519.log 2024-05-08T01:02:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.84352.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.84352.log.gz 2024-05-08T01:02:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose 88.5% -- -- replaced with /var/log/ceph/cluster2-client.mirror.47565.log.gz 2024-05-08T01:02:07.924 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/cluster2-client.mirror.92162.log 2024-05-08T01:02:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49260.log 2024-05-08T01:02:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92162.log: /var/log/ceph/cluster1-client.mirror.89081.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89081.log.gz 2024-05-08T01:02:07.926 INFO:teuthology.orchestra.run.smithi116.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92162.log.gz 2024-05-08T01:02:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56242.log 2024-05-08T01:02:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.49260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73280.log 2024-05-08T01:02:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.56242.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51268.log.gz 2024-05-08T01:02:07.926 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39511.log.gz 2024-05-08T01:02:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71098.log 2024-05-08T01:02:07.929 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42685.log.gz 2024-05-08T01:02:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.73280.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49519.log.gz 2024-05-08T01:02:07.930 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73280.log.gz 2024-05-08T01:02:07.932 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43981.log.gz 2024-05-08T01:02:07.935 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.admin.49260.log.gz 2024-05-08T01:02:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75817.log 2024-05-08T01:02:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.71098.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71098.log.gz 2024-05-08T01:02:07.951 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53929.log.gz 2024-05-08T01:02:07.951 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43702.log 2024-05-08T01:02:07.951 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.75817.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56529.log.gz 2024-05-08T01:02:07.952 INFO:teuthology.orchestra.run.smithi116.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75817.log.gz 2024-05-08T01:02:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90386.log 2024-05-08T01:02:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56506.log 2024-05-08T01:02:07.957 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37611.log 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.90386.log.gz 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.56506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91663.log 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56506.log.gz 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.37611.log: -- /var/log/ceph/cluster2-client.mirror.48802.log 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94218.log 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91663.log.gz 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67339.log 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.94218.log: /var/log/ceph/cluster2-client.mirror.76981.log 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.67339.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94218.log.gz 2024-05-08T01:02:07.958 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67339.log.gz 2024-05-08T01:02:07.962 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43702.log.gz 2024-05-08T01:02:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.94911.log 2024-05-08T01:02:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.76981.log: /var/log/ceph/cluster1-client.mirror.39035.log 2024-05-08T01:02:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36192.log 2024-05-08T01:02:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.39035.log: /var/log/ceph/cluster2-client.admin.94911.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.94911.log.gz 2024-05-08T01:02:07.962 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48802.log.gz 2024-05-08T01:02:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91817.log 2024-05-08T01:02:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36192.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36192.log.gz 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39035.log.gz 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93161.log 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42798.log 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76981.log.gz 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93161.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.91817.log.gz 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93161.log.gz 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89419.log 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51787.log 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89419.log: /var/log/ceph/cluster1-client.mirror.42798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73977.log 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89419.log.gz 2024-05-08T01:02:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83227.log 2024-05-08T01:02:07.969 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91151.log 2024-05-08T01:02:07.969 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.73977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50360.log 2024-05-08T01:02:07.969 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73977.log.gz 2024-05-08T01:02:07.969 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89756.log 2024-05-08T01:02:07.969 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50360.log: /var/log/ceph/cluster1-client.mirror.91151.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42798.log.gz 2024-05-08T01:02:07.969 INFO:teuthology.orchestra.run.smithi064.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91151.log.gz 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83227.log.gz 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47038.log 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67774.log 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47038.log: /var/log/ceph/cluster1-client.mirror.89756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95163.log 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89756.log.gz 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.67774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48096.log 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.95163.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.95163.log.gz 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44729.log 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35349.log 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44729.log: /var/log/ceph/cluster1-client.mirror.48096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41565.log 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51787.log.gz 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35349.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35349.log.gz 2024-05-08T01:02:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84011.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67774.log.gz 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47321.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.84011.log: --verbose -- /var/log/ceph/cluster2-client.mirror.85045.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68446.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.85045.log: -- /var/log/ceph/cluster2-client.mirror.39149.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.68446.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85045.log.gz 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68446.log.gz 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89126.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51113.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.84011.log.gz 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89126.log: /var/log/ceph/cluster2-client.mirror.39149.log: gzip 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89126.log.gz 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48443.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51113.log: /var/log/ceph/cluster2-client.mirror.48443.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74776.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85577.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87330.log 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47038.log.gz 2024-05-08T01:02:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74776.log: /var/log/ceph/cluster2-client.mirror.85577.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74776.log.gz 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47321.log.gz 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85577.log.gz 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76017.log 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86266.log 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.76017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64568.log 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86266.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87330.log.gz 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92922.log 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.64568.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86266.log.gz 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51113.log.gz 2024-05-08T01:02:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77832.log 2024-05-08T01:02:07.993 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92922.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41565.log.gz 2024-05-08T01:02:07.993 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.92922.log.gz 2024-05-08T01:02:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43795.log 2024-05-08T01:02:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77832.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77832.log.gz 2024-05-08T01:02:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35223.log 2024-05-08T01:02:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43795.log: 95.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77545.log 2024-05-08T01:02:07.994 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53731.log.gz 2024-05-08T01:02:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35223.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35223.log.gz 2024-05-08T01:02:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49234.log 2024-05-08T01:02:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41100.log 2024-05-08T01:02:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49234.log: /var/log/ceph/cluster2-client.mirror.77545.log: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42908.log.gz 2024-05-08T01:02:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66221.log 2024-05-08T01:02:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41100.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93503.log 2024-05-08T01:02:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.66221.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66221.log.gz 2024-05-08T01:02:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73725.log 2024-05-08T01:02:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93503.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41100.log.gz 2024-05-08T01:02:08.005 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43795.log.gz 2024-05-08T01:02:08.007 INFO:teuthology.orchestra.run.smithi116.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93503.log.gz 2024-05-08T01:02:08.007 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.44729.log.gz 2024-05-08T01:02:08.007 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66674.log 2024-05-08T01:02:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.73725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40972.log 2024-05-08T01:02:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.66674.log: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39149.log.gz 2024-05-08T01:02:08.026 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48096.log.gz 2024-05-08T01:02:08.028 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80116.log.gz 2024-05-08T01:02:08.036 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78335.log 2024-05-08T01:02:08.050 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50739.log 2024-05-08T01:02:08.052 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.78335.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78335.log.gz 2024-05-08T01:02:08.054 INFO:teuthology.orchestra.run.smithi116.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50360.log.gz 2024-05-08T01:02:08.065 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50606.log 2024-05-08T01:02:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50739.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37611.log.gz 2024-05-08T01:02:08.078 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72894.log 2024-05-08T01:02:08.087 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50606.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39060.log 2024-05-08T01:02:08.087 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.72894.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.72894.log.gz 2024-05-08T01:02:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59287.log 2024-05-08T01:02:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39060.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66674.log.gz 2024-05-08T01:02:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50317.log 2024-05-08T01:02:08.116 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.59287.log: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59891.log.gz 2024-05-08T01:02:08.122 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37892.log 2024-05-08T01:02:08.123 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50317.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88892.log 2024-05-08T01:02:08.127 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68539.log 2024-05-08T01:02:08.128 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88892.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88892.log.gz 2024-05-08T01:02:08.130 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37892.log.gz 2024-05-08T01:02:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94011.log 2024-05-08T01:02:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.68539.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68539.log.gz 2024-05-08T01:02:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38701.log 2024-05-08T01:02:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.94011.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94011.log.gz 2024-05-08T01:02:08.135 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75597.log 2024-05-08T01:02:08.135 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.38701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89644.log 2024-05-08T01:02:08.137 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75597.log: 95.5% 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75597.log.gz 2024-05-08T01:02:08.137 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.76017.log.gz 2024-05-08T01:02:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48709.log 2024-05-08T01:02:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95079.log 2024-05-08T01:02:08.140 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89644.log.gz 2024-05-08T01:02:08.140 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79645.log 2024-05-08T01:02:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.95079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50782.log 2024-05-08T01:02:08.141 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.95079.log.gz 2024-05-08T01:02:08.141 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72063.log 2024-05-08T01:02:08.144 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70401.log 2024-05-08T01:02:08.144 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72063.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72063.log.gz 2024-05-08T01:02:08.144 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53170.log 2024-05-08T01:02:08.146 INFO:teuthology.orchestra.run.smithi116.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79645.log.gz 2024-05-08T01:02:08.146 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48709.log.gz 2024-05-08T01:02:08.146 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45194.log 2024-05-08T01:02:08.146 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53170.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53170.log.gz 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.70401.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70401.log.gz 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57320.log.gz 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90364.log 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80858.log 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81510.log 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90364.log.gz 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.80858.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86246.log 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90656.log 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86246.log: /var/log/ceph/cluster2-client.mirror.81510.log: 88.2% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86246.log.gz 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.81510.log.gz 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35478.log 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.80858.log.gz 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57297.log 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35478.log: /var/log/ceph/cluster1-client.mirror.90656.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.35478.log.gz 2024-05-08T01:02:08.167 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81244.log 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.90656.log.gz/var/log/ceph/cluster2-client.mirror.57297.log: 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57297.log.gz 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.92110.log 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.81244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35898.log 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92110.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92110.log.gz 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59501.log 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43574.log 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.35898.log: 92.1%/var/log/ceph/cluster2-client.mirror.59501.log: -- replaced with /var/log/ceph/cluster1-client.mirror.81244.log.gz 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35898.log.gz 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36700.log 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93140.log 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36700.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92339.log 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91286.log 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.93140.log.gz 2024-05-08T01:02:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47379.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91286.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92339.log.gz 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91286.log.gz 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48008.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40503.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.48008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42092.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55494.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67749.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.55494.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45194.log.gz 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47379.log.gz 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/cluster2-client.mirror.55471.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.67749.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67749.log.gz 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51684.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.55471.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55471.log.gz 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83521.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67396.log 2024-05-08T01:02:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83521.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83521.log.gz 2024-05-08T01:02:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48208.log 2024-05-08T01:02:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.67396.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67396.log.gz 2024-05-08T01:02:08.171 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48443.log.gz 2024-05-08T01:02:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53276.log 2024-05-08T01:02:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.48208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75883.log 2024-05-08T01:02:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53276.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53276.log.gz 2024-05-08T01:02:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39315.log 2024-05-08T01:02:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75883.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75883.log.gz 2024-05-08T01:02:08.175 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68160.log 2024-05-08T01:02:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39315.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42092.log.gz 2024-05-08T01:02:08.178 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40503.log.gz 2024-05-08T01:02:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94360.log 2024-05-08T01:02:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.68160.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68160.log.gz 2024-05-08T01:02:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88130.log 2024-05-08T01:02:08.185 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48008.log.gz 2024-05-08T01:02:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94360.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56242.log.gz 2024-05-08T01:02:08.185 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94360.log.gz 2024-05-08T01:02:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43574.log.gz 2024-05-08T01:02:08.185 INFO:teuthology.orchestra.run.smithi116.stderr: 80.1% -- /var/log/ceph/cluster1-client.mirror.93408.log 2024-05-08T01:02:08.186 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.36700.log.gz 2024-05-08T01:02:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94158.log 2024-05-08T01:02:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88130.log: /var/log/ceph/cluster1-client.mirror.93408.log: 88.6% 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88130.log.gz 2024-05-08T01:02:08.186 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.48208.log.gz 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93408.log.gz 2024-05-08T01:02:08.186 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89554.log 2024-05-08T01:02:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94158.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94158.log.gz 2024-05-08T01:02:08.186 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44132.log 2024-05-08T01:02:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91398.log 2024-05-08T01:02:08.190 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89554.log.gz 2024-05-08T01:02:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57609.log 2024-05-08T01:02:08.190 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72098.log 2024-05-08T01:02:08.190 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.91398.log.gz 2024-05-08T01:02:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.57609.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57609.log.gz 2024-05-08T01:02:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43419.log 2024-05-08T01:02:08.191 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50782.log.gz 2024-05-08T01:02:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72098.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44132.log.gz 2024-05-08T01:02:08.191 INFO:teuthology.orchestra.run.smithi116.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39060.log.gz 2024-05-08T01:02:08.191 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72098.log.gz 2024-05-08T01:02:08.191 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36461.log 2024-05-08T01:02:08.193 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40879.log 2024-05-08T01:02:08.193 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75975.log 2024-05-08T01:02:08.193 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40879.log: /var/log/ceph/cluster1-client.admin.36461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91683.log 2024-05-08T01:02:08.193 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86808.log 2024-05-08T01:02:08.194 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91683.log: /var/log/ceph/cluster2-client.mirror.75975.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40972.log.gz 2024-05-08T01:02:08.195 INFO:teuthology.orchestra.run.smithi116.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.36461.log.gz 2024-05-08T01:02:08.195 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75975.log.gz 2024-05-08T01:02:08.195 INFO:teuthology.orchestra.run.smithi116.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91683.log.gz 2024-05-08T01:02:08.195 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91039.log 2024-05-08T01:02:08.196 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83247.log 2024-05-08T01:02:08.196 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91039.log: /var/log/ceph/cluster2-client.mirror.86808.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.91039.log.gz 2024-05-08T01:02:08.197 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86808.log.gz 2024-05-08T01:02:08.197 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43516.log 2024-05-08T01:02:08.199 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.83247.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.83247.log.gz 2024-05-08T01:02:08.200 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78402.log 2024-05-08T01:02:08.200 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43419.log.gz 2024-05-08T01:02:08.201 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51550.log 2024-05-08T01:02:08.203 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.78402.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43516.log.gz 2024-05-08T01:02:08.204 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47657.log 2024-05-08T01:02:08.207 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86950.log 2024-05-08T01:02:08.208 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83655.log 2024-05-08T01:02:08.211 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86950.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86950.log.gz 2024-05-08T01:02:08.211 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93543.log 2024-05-08T01:02:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83655.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77545.log.gz 2024-05-08T01:02:08.216 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50227.log 2024-05-08T01:02:08.218 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93543.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.93543.log.gz 2024-05-08T01:02:08.219 INFO:teuthology.orchestra.run.smithi116.stderr: 96.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83655.log.gz 2024-05-08T01:02:08.219 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80368.log 2024-05-08T01:02:08.219 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69477.log 2024-05-08T01:02:08.220 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80368.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.80368.log.gz 2024-05-08T01:02:08.220 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45438.log 2024-05-08T01:02:08.221 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69477.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69477.log.gz 2024-05-08T01:02:08.221 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89509.log 2024-05-08T01:02:08.224 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91241.log 2024-05-08T01:02:08.224 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89509.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89509.log.gz 2024-05-08T01:02:08.224 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36437.log 2024-05-08T01:02:08.224 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91241.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91241.log.gz 2024-05-08T01:02:08.227 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84925.log 2024-05-08T01:02:08.227 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.36437.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36437.log.gz 2024-05-08T01:02:08.227 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45438.log.gz 2024-05-08T01:02:08.227 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.77478.log 2024-05-08T01:02:08.227 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84925.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84925.log.gz 2024-05-08T01:02:08.227 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36768.log 2024-05-08T01:02:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.77478.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77478.log.gz 2024-05-08T01:02:08.230 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42371.log 2024-05-08T01:02:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90274.log 2024-05-08T01:02:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.36768.log: /var/log/ceph/cluster2-client.mirror.42371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43384.log 2024-05-08T01:02:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90274.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90274.log.gz 2024-05-08T01:02:08.231 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87310.log 2024-05-08T01:02:08.233 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43384.log: 93.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92024.log 2024-05-08T01:02:08.233 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87310.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87310.log.gz 2024-05-08T01:02:08.233 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79067.log 2024-05-08T01:02:08.233 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50739.log.gz 2024-05-08T01:02:08.234 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92024.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92024.log.gz 2024-05-08T01:02:08.234 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80074.log 2024-05-08T01:02:08.234 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50317.log.gz 2024-05-08T01:02:08.235 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79067.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79067.log.gz 2024-05-08T01:02:08.235 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38681.log 2024-05-08T01:02:08.236 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36529.log 2024-05-08T01:02:08.238 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38681.log: /var/log/ceph/cluster2-client.mirror.80074.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80074.log.gz 2024-05-08T01:02:08.238 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65190.log 2024-05-08T01:02:08.239 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.36529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69134.log 2024-05-08T01:02:08.239 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.65190.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65190.log.gz 2024-05-08T01:02:08.241 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34970.log 2024-05-08T01:02:08.241 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69134.log: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36768.log.gz 2024-05-08T01:02:08.242 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69134.log.gz 2024-05-08T01:02:08.242 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89060.log 2024-05-08T01:02:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34970.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34970.log.gz 2024-05-08T01:02:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70248.log 2024-05-08T01:02:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89060.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.89060.log.gz 2024-05-08T01:02:08.244 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85386.log 2024-05-08T01:02:08.244 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42371.log.gz 2024-05-08T01:02:08.245 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.70248.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43384.log.gz 2024-05-08T01:02:08.245 INFO:teuthology.orchestra.run.smithi116.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70248.log.gz 2024-05-08T01:02:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54985.log 2024-05-08T01:02:08.246 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85386.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85386.log.gz 2024-05-08T01:02:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31544.log 2024-05-08T01:02:08.247 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.54985.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54985.log.gz 2024-05-08T01:02:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93264.log 2024-05-08T01:02:08.248 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.31544.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31544.log.gz 2024-05-08T01:02:08.249 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93290.log 2024-05-08T01:02:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93264.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93264.log.gz 2024-05-08T01:02:08.250 INFO:teuthology.orchestra.run.smithi116.stderr: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59287.log.gz 2024-05-08T01:02:08.250 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55248.log 2024-05-08T01:02:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93290.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.36529.log.gz 2024-05-08T01:02:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84397.log 2024-05-08T01:02:08.257 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68135.log 2024-05-08T01:02:08.257 INFO:teuthology.orchestra.run.smithi116.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51550.log.gz 2024-05-08T01:02:08.258 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.55248.log: /var/log/ceph/cluster1-client.mirror.84397.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50606.log.gz 2024-05-08T01:02:08.259 INFO:teuthology.orchestra.run.smithi116.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.84397.log.gz 2024-05-08T01:02:08.260 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89171.log 2024-05-08T01:02:08.260 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.68135.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68135.log.gz 2024-05-08T01:02:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35097.log 2024-05-08T01:02:08.261 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89171.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89171.log.gz 2024-05-08T01:02:08.262 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94117.log 2024-05-08T01:02:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35097.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35097.log.gz 2024-05-08T01:02:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91964.log 2024-05-08T01:02:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94117.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.94117.log.gz 2024-05-08T01:02:08.263 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36751.log 2024-05-08T01:02:08.264 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91964.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91964.log.gz 2024-05-08T01:02:08.265 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71776.log 2024-05-08T01:02:08.265 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.36751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89486.log 2024-05-08T01:02:08.268 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.71776.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71776.log.gz 2024-05-08T01:02:08.268 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65225.log 2024-05-08T01:02:08.268 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89486.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89486.log.gz 2024-05-08T01:02:08.268 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40879.log.gz 2024-05-08T01:02:08.269 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81988.log 2024-05-08T01:02:08.269 INFO:teuthology.orchestra.run.smithi116.stderr: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36751.log.gz 2024-05-08T01:02:08.269 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.65225.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65225.log.gz 2024-05-08T01:02:08.270 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70812.log 2024-05-08T01:02:08.272 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81988.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81988.log.gz 2024-05-08T01:02:08.272 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51419.log 2024-05-08T01:02:08.275 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.70812.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70812.log.gz 92.7% 2024-05-08T01:02:08.275 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38681.log.gz 2024-05-08T01:02:08.275 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92317.log 2024-05-08T01:02:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86530.log 2024-05-08T01:02:08.277 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92317.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51419.log.gz 2024-05-08T01:02:08.277 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92317.log.gz 2024-05-08T01:02:08.279 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64568.log.gz 2024-05-08T01:02:08.284 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38701.log.gz 2024-05-08T01:02:08.286 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47414.log 2024-05-08T01:02:08.287 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86530.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86530.log.gz 2024-05-08T01:02:08.287 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94076.log 2024-05-08T01:02:08.288 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77889.log 2024-05-08T01:02:08.290 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94076.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94076.log.gz 2024-05-08T01:02:08.290 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89711.log 2024-05-08T01:02:08.290 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47414.log.gz 2024-05-08T01:02:08.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61767.log 2024-05-08T01:02:08.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89711.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77889.log.gz 2024-05-08T01:02:08.293 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87741.log 2024-05-08T01:02:08.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.61767.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89711.log.gz 2024-05-08T01:02:08.329 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55494.log.gz 2024-05-08T01:02:08.331 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90994.log 2024-05-08T01:02:08.331 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87741.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87741.log.gz 2024-05-08T01:02:08.332 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88081.log 2024-05-08T01:02:08.332 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90994.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90994.log.gz 2024-05-08T01:02:08.335 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35814.log 2024-05-08T01:02:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88081.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88081.log.gz 2024-05-08T01:02:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80813.log 2024-05-08T01:02:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.35814.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35814.log.gz 2024-05-08T01:02:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92439.log 2024-05-08T01:02:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.80813.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80813.log.gz 2024-05-08T01:02:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47800.log 2024-05-08T01:02:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92439.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92439.log.gz 2024-05-08T01:02:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38793.log 2024-05-08T01:02:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.47800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79623.log 2024-05-08T01:02:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89239.log 2024-05-08T01:02:08.343 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.79623.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79623.log.gz 2024-05-08T01:02:08.343 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31918.log 2024-05-08T01:02:08.343 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.47800.log.gz 2024-05-08T01:02:08.343 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89239.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89239.log.gz 2024-05-08T01:02:08.343 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38136.log 2024-05-08T01:02:08.343 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.31918.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31918.log.gz 2024-05-08T01:02:08.343 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43140.log 2024-05-08T01:02:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38136.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81545.log 2024-05-08T01:02:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93565.log 2024-05-08T01:02:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81545.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81545.log.gz 2024-05-08T01:02:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86317.log 2024-05-08T01:02:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93565.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93565.log.gz 2024-05-08T01:02:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93441.log 2024-05-08T01:02:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86317.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86317.log.gz 2024-05-08T01:02:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41251.log 2024-05-08T01:02:08.347 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38136.log.gz 2024-05-08T01:02:08.348 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43140.log.gz 2024-05-08T01:02:08.348 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93441.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.93441.log.gz 2024-05-08T01:02:08.350 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78210.log 2024-05-08T01:02:08.350 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66607.log 2024-05-08T01:02:08.350 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.78210.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78210.log.gz 2024-05-08T01:02:08.350 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52700.log 2024-05-08T01:02:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.66607.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66607.log.gz 2024-05-08T01:02:08.355 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38841.log 2024-05-08T01:02:08.355 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48988.log 2024-05-08T01:02:08.355 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.38841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74808.log 2024-05-08T01:02:08.355 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73725.log.gz 2024-05-08T01:02:08.355 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95247.log 2024-05-08T01:02:08.359 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.74808.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74808.log.gz 2024-05-08T01:02:08.359 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41251.log.gz 2024-05-08T01:02:08.359 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43888.log 2024-05-08T01:02:08.359 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.95247.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.95247.log.gz 2024-05-08T01:02:08.359 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95205.log 2024-05-08T01:02:08.359 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31757.log 2024-05-08T01:02:08.361 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.38841.log.gz 2024-05-08T01:02:08.361 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.95205.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.95205.log.gz 2024-05-08T01:02:08.362 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43888.log.gz 2024-05-08T01:02:08.362 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65257.log 2024-05-08T01:02:08.362 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92619.log 2024-05-08T01:02:08.362 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.31757.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31757.log.gz 2024-05-08T01:02:08.371 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.65257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80017.log 2024-05-08T01:02:08.371 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92619.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65257.log.gz 2024-05-08T01:02:08.371 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.92619.log.gz 2024-05-08T01:02:08.371 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31856.log 2024-05-08T01:02:08.371 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80017.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48988.log.gz 2024-05-08T01:02:08.371 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80017.log.gz 2024-05-08T01:02:08.371 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40821.log 2024-05-08T01:02:08.371 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.31856.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31856.log.gz 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53698.log 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.40821.log: --verbose -- /var/log/ceph/cluster2-client.mirror.82660.log 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36683.log 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82660.log: /var/log/ceph/cluster2-client.mirror.53698.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53698.log.gz 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.82660.log.gz 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64159.log 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65611.log 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.64159.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64159.log.gz 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61983.log 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90296.log 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.61983.log: /var/log/ceph/cluster2-client.mirror.65611.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61983.log.gz 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65611.log.gz 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67438.log 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79192.log 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90296.log: /var/log/ceph/cluster2-client.mirror.67438.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90296.log.gz 2024-05-08T01:02:08.372 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41193.log 2024-05-08T01:02:08.373 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.79192.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79192.log.gz 2024-05-08T01:02:08.374 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46480.log 2024-05-08T01:02:08.374 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40038.log 2024-05-08T01:02:08.376 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46480.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.36683.log.gz 2024-05-08T01:02:08.376 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39172.log 2024-05-08T01:02:08.376 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74044.log 2024-05-08T01:02:08.377 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73347.log 2024-05-08T01:02:08.377 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.74044.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74044.log.gz 2024-05-08T01:02:08.380 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76739.log 2024-05-08T01:02:08.382 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.73347.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41193.log.gz 2024-05-08T01:02:08.384 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40038.log.gz 2024-05-08T01:02:08.394 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45756.log 2024-05-08T01:02:08.396 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.76739.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76739.log.gz 2024-05-08T01:02:08.409 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49991.log 2024-05-08T01:02:08.417 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46631.log 2024-05-08T01:02:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49991.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45756.log.gz 2024-05-08T01:02:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90679.log 2024-05-08T01:02:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46631.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49991.log.gz 2024-05-08T01:02:08.434 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59868.log 2024-05-08T01:02:08.435 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90679.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90679.log.gz 2024-05-08T01:02:08.439 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47657.log.gz 2024-05-08T01:02:08.442 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87619.log 2024-05-08T01:02:08.442 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.59868.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59868.log.gz 2024-05-08T01:02:08.443 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46480.log.gz 2024-05-08T01:02:08.443 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46631.log.gz 2024-05-08T01:02:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88343.log 2024-05-08T01:02:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87619.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87619.log.gz 2024-05-08T01:02:08.460 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61767.log.gz 2024-05-08T01:02:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85846.log 2024-05-08T01:02:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88343.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93290.log.gz 2024-05-08T01:02:08.464 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88343.log.gz 2024-05-08T01:02:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55646.log 2024-05-08T01:02:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77797.log 2024-05-08T01:02:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.55646.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85846.log.gz 2024-05-08T01:02:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39945.log 2024-05-08T01:02:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.77797.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77797.log.gz 2024-05-08T01:02:08.468 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93462.log 2024-05-08T01:02:08.469 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86737.log 2024-05-08T01:02:08.471 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39945.log.gz 2024-05-08T01:02:08.473 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93462.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52700.log.gz 2024-05-08T01:02:08.473 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93462.log.gz 2024-05-08T01:02:08.473 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34592.log 2024-05-08T01:02:08.477 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86737.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.86737.log.gz 2024-05-08T01:02:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53002.log 2024-05-08T01:02:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34592.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34592.log.gz 2024-05-08T01:02:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84854.log 2024-05-08T01:02:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53002.log: 82.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53002.log.gz 2024-05-08T01:02:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35350.log 2024-05-08T01:02:08.480 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84854.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84854.log.gz 2024-05-08T01:02:08.481 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51180.log 2024-05-08T01:02:08.482 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35350.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35350.log.gz 2024-05-08T01:02:08.485 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73248.log 2024-05-08T01:02:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51180.log: gzip 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40821.log.gz 2024-05-08T01:02:08.486 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.36478.log 2024-05-08T01:02:08.488 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.73248.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73248.log.gz 2024-05-08T01:02:08.488 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67416.log 2024-05-08T01:02:08.488 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.36478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93887.log 2024-05-08T01:02:08.489 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.67416.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67416.log.gz 2024-05-08T01:02:08.493 INFO:teuthology.orchestra.run.smithi116.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.36478.log.gz 2024-05-08T01:02:08.494 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51180.log.gz 2024-05-08T01:02:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37586.log 2024-05-08T01:02:08.505 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93887.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78402.log.gz 2024-05-08T01:02:08.507 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47135.log 2024-05-08T01:02:08.512 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.37586.log: /var/log/ceph/cluster2-client.mirror.47135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90746.log 2024-05-08T01:02:08.512 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78654.log 2024-05-08T01:02:08.512 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90746.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90746.log.gz 2024-05-08T01:02:08.513 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37586.log.gz 2024-05-08T01:02:08.516 INFO:teuthology.orchestra.run.smithi116.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59501.log.gz 2024-05-08T01:02:08.516 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47135.log.gz 2024-05-08T01:02:08.516 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36512.log 2024-05-08T01:02:08.517 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.78654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51487.log 2024-05-08T01:02:08.517 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78654.log.gz 2024-05-08T01:02:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51487.log: /var/log/ceph/cluster1-client.mirror.36512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43037.log 2024-05-08T01:02:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87288.log 2024-05-08T01:02:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86788.log 2024-05-08T01:02:08.521 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87288.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87288.log.gz 2024-05-08T01:02:08.521 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94136.log 2024-05-08T01:02:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86788.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86788.log.gz 2024-05-08T01:02:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65688.log 2024-05-08T01:02:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94136.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.94136.log.gz 2024-05-08T01:02:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51487.log.gz 2024-05-08T01:02:08.526 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/cluster2-client.admin.31717.log 2024-05-08T01:02:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38547.log 2024-05-08T01:02:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.31717.log: /var/log/ceph/cluster1-client.mirror.65688.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31717.log.gz 2024-05-08T01:02:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95289.log/var/log/ceph/cluster2-client.mirror.38547.log: 54.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36512.log.gz 2024-05-08T01:02:08.526 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.95289.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.95289.log.gz 2024-05-08T01:02:08.529 INFO:teuthology.orchestra.run.smithi116.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65688.log.gz 2024-05-08T01:02:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51530.log 2024-05-08T01:02:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49754.log 2024-05-08T01:02:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80502.log 2024-05-08T01:02:08.530 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.49754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88421.log 2024-05-08T01:02:08.530 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80502.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38547.log.gz 2024-05-08T01:02:08.532 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51530.log.gz 2024-05-08T01:02:08.533 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91642.log 2024-05-08T01:02:08.536 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88421.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88421.log.gz 2024-05-08T01:02:08.536 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90499.log 2024-05-08T01:02:08.536 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73347.log.gz 2024-05-08T01:02:08.536 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91642.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91642.log.gz 2024-05-08T01:02:08.538 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89779.log 2024-05-08T01:02:08.538 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.90499.log: -- /var/log/ceph/cluster1-client.mirror.50936.log 2024-05-08T01:02:08.538 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89779.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89779.log.gz 2024-05-08T01:02:08.539 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42762.log 2024-05-08T01:02:08.540 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50936.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61081.log 2024-05-08T01:02:08.543 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42762.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42762.log.gz 2024-05-08T01:02:08.544 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90499.log.gz 2024-05-08T01:02:08.547 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38589.log 2024-05-08T01:02:08.547 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.61081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87690.log 2024-05-08T01:02:08.547 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61081.log.gz 2024-05-08T01:02:08.547 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80460.log 2024-05-08T01:02:08.547 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81577.log 2024-05-08T01:02:08.549 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80460.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.80460.log.gz 2024-05-08T01:02:08.551 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87690.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87690.log.gz 2024-05-08T01:02:08.555 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56678.log 2024-05-08T01:02:08.555 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.81577.log: -- replaced with /var/log/ceph/cluster2-client.mirror.38589.log.gz 2024-05-08T01:02:08.555 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42866.log 2024-05-08T01:02:08.556 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81577.log.gz 2024-05-08T01:02:08.556 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.56678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78268.log 2024-05-08T01:02:08.556 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42866.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56678.log.gz 2024-05-08T01:02:08.556 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72583.log 2024-05-08T01:02:08.556 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.78268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94623.log 2024-05-08T01:02:08.556 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78268.log.gz 2024-05-08T01:02:08.556 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84320.log 2024-05-08T01:02:08.557 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67438.log.gz 2024-05-08T01:02:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.94623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94623.log.gz 2024-05-08T01:02:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45721.log 2024-05-08T01:02:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84320.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84320.log.gz 2024-05-08T01:02:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93483.log 2024-05-08T01:02:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50181.log 2024-05-08T01:02:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55225.log 2024-05-08T01:02:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50181.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.93483.log.gz 2024-05-08T01:02:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89329.log 2024-05-08T01:02:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.55225.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55225.log.gz 2024-05-08T01:02:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36717.log 2024-05-08T01:02:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89329.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89329.log.gz 2024-05-08T01:02:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76959.log 2024-05-08T01:02:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94381.log 2024-05-08T01:02:08.565 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.76959.log: /var/log/ceph/cluster2-client.mirror.84732.log 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94381.log: 88.3% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45721.log.gz 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.94381.log.gz 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38900.log 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76959.log.gz 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53906.log 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45933.log 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53906.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53906.log.gz 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90094.log 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45933.log: 70.0% -- replaced with /var/log/ceph/cluster2-client.admin.36717.log.gz 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50181.log.gz 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84732.log.gz 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51752.log 2024-05-08T01:02:08.573 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90094.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90094.log.gz 2024-05-08T01:02:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89666.log 2024-05-08T01:02:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51752.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67304.log 2024-05-08T01:02:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48053.log 2024-05-08T01:02:08.574 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89666.log.gz 2024-05-08T01:02:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.67304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38229.log 2024-05-08T01:02:08.574 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.48053.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67304.log.gz 2024-05-08T01:02:08.576 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53193.log 2024-05-08T01:02:08.576 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.94599.log 2024-05-08T01:02:08.576 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94270.log 2024-05-08T01:02:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.94599.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.94599.log.gz 2024-05-08T01:02:08.580 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92046.log 2024-05-08T01:02:08.583 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94270.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38229.log.gz 2024-05-08T01:02:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.91367.log 2024-05-08T01:02:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92046.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94270.log.gz 2024-05-08T01:02:08.591 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92046.log.gz 2024-05-08T01:02:08.591 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93524.log 95.7% 2024-05-08T01:02:08.592 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.80502.log.gz 2024-05-08T01:02:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.0.91367.log: 21.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.91367.log.gz 2024-05-08T01:02:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58713.log 2024-05-08T01:02:08.592 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36066.log 2024-05-08T01:02:08.592 INFO:teuthology.orchestra.run.smithi116.stderr: 83.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93524.log.gz 2024-05-08T01:02:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.58713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84465.log 2024-05-08T01:02:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36066.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36066.log.gz 2024-05-08T01:02:08.594 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87599.log 2024-05-08T01:02:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84465.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84465.log.gz 2024-05-08T01:02:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35310.log 2024-05-08T01:02:08.597 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87599.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87599.log.gz 2024-05-08T01:02:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85335.log 2024-05-08T01:02:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35310.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35310.log.gz 2024-05-08T01:02:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53025.log 2024-05-08T01:02:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85335.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85335.log.gz 2024-05-08T01:02:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65282.log 2024-05-08T01:02:08.609 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53025.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55248.log.gz 2024-05-08T01:02:08.613 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.65282.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65282.log.gz 2024-05-08T01:02:08.613 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88614.log 2024-05-08T01:02:08.613 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88563.log 2024-05-08T01:02:08.615 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88614.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88614.log.gz 2024-05-08T01:02:08.615 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44074.log 2024-05-08T01:02:08.617 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88563.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88563.log.gz 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42866.log.gz 2024-05-08T01:02:08.617 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.617 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52476.log 2024-05-08T01:02:08.619 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48396.log 2024-05-08T01:02:08.619 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50825.log 2024-05-08T01:02:08.620 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87430.log 2024-05-08T01:02:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51814.log 2024-05-08T01:02:08.622 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87430.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87430.log.gz 2024-05-08T01:02:08.624 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93839.log 2024-05-08T01:02:08.624 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46066.log 2024-05-08T01:02:08.625 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93839.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.93839.log.gz 2024-05-08T01:02:08.625 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70546.log 2024-05-08T01:02:08.626 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.46066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37789.log 2024-05-08T01:02:08.627 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.70546.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44074.log.gz 2024-05-08T01:02:08.628 INFO:teuthology.orchestra.run.smithi116.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70546.log.gz 2024-05-08T01:02:08.628 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70290.log 2024-05-08T01:02:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88201.log 2024-05-08T01:02:08.637 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.70290.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45933.log.gz 2024-05-08T01:02:08.650 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86059.log 2024-05-08T01:02:08.651 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88201.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88201.log.gz 2024-05-08T01:02:08.652 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50936.log.gz 2024-05-08T01:02:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45287.log 2024-05-08T01:02:08.667 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86059.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86059.log.gz 2024-05-08T01:02:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42034.log 2024-05-08T01:02:08.689 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47704.log 2024-05-08T01:02:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42034.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70290.log.gz 2024-05-08T01:02:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48177.log 2024-05-08T01:02:08.701 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51951.log 2024-05-08T01:02:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.48177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90881.log 2024-05-08T01:02:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91984.log 2024-05-08T01:02:08.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90881.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.90881.log.gz 2024-05-08T01:02:08.706 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85315.log 2024-05-08T01:02:08.708 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91984.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91984.log.gz 2024-05-08T01:02:08.709 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44694.log 2024-05-08T01:02:08.709 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85315.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85315.log.gz 2024-05-08T01:02:08.711 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72484.log 2024-05-08T01:02:08.711 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36853.log 2024-05-08T01:02:08.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72484.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72583.log.gz 2024-05-08T01:02:08.715 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72484.log.gz 2024-05-08T01:02:08.715 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86930.log 2024-05-08T01:02:08.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.36853.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36853.log.gz 2024-05-08T01:02:08.715 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76682.log 2024-05-08T01:02:08.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82309.log 2024-05-08T01:02:08.716 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86930.log.gz 2024-05-08T01:02:08.716 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.76682.log: gzip 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76682.log.gz 2024-05-08T01:02:08.716 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82891.log 2024-05-08T01:02:08.716 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36939.log 2024-05-08T01:02:08.716 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82309.log.gz 2024-05-08T01:02:08.717 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69837.log 2024-05-08T01:02:08.718 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.36939.log: /var/log/ceph/cluster2-client.mirror.82891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38744.log 2024-05-08T01:02:08.720 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.69837.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69837.log.gz 2024-05-08T01:02:08.720 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60287.log 2024-05-08T01:02:08.721 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.38744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43105.log 2024-05-08T01:02:08.721 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44694.log.gz 2024-05-08T01:02:08.722 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.60287.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45287.log.gz 2024-05-08T01:02:08.723 INFO:teuthology.orchestra.run.smithi116.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82891.log.gz 2024-05-08T01:02:08.723 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87050.log 2024-05-08T01:02:08.725 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52566.log 2024-05-08T01:02:08.725 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.46066.log.gz 2024-05-08T01:02:08.725 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87050.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.admin.48177.log.gz 2024-05-08T01:02:08.725 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87050.log.gz 2024-05-08T01:02:08.725 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88450.log 2024-05-08T01:02:08.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52566.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81266.log 2024-05-08T01:02:08.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88450.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88450.log.gz 2024-05-08T01:02:08.728 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68481.log 2024-05-08T01:02:08.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81266.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42034.log.gz 2024-05-08T01:02:08.730 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43105.log.gz 2024-05-08T01:02:08.740 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49097.log 2024-05-08T01:02:08.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.68481.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68481.log.gz 2024-05-08T01:02:08.748 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55646.log.gz 2024-05-08T01:02:08.749 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70426.log 2024-05-08T01:02:08.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.49097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56825.log 2024-05-08T01:02:08.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.70426.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70426.log.gz 2024-05-08T01:02:08.752 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74833.log 2024-05-08T01:02:08.766 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.56825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48463.log 2024-05-08T01:02:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74833.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74833.log.gz 2024-05-08T01:02:08.769 INFO:teuthology.orchestra.run.smithi116.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52476.log.gz 2024-05-08T01:02:08.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49097.log.gz 2024-05-08T01:02:08.778 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58713.log.gz 2024-05-08T01:02:08.784 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50227.log.gz 2024-05-08T01:02:08.785 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93659.log 2024-05-08T01:02:08.788 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.48463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.36939.log 2024-05-08T01:02:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93659.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93659.log.gz 2024-05-08T01:02:08.789 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39256.log 2024-05-08T01:02:08.790 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.1.36939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80049.log 2024-05-08T01:02:08.791 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.39256.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50383.log 2024-05-08T01:02:08.793 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.80049.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80049.log.gz 2024-05-08T01:02:08.793 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70755.log 2024-05-08T01:02:08.793 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.81266.log.gz 2024-05-08T01:02:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48145.log 2024-05-08T01:02:08.795 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.70755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89959.log 2024-05-08T01:02:08.796 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8%/var/log/ceph/cluster2-client.mirror.48145.log: -- replaced with /var/log/ceph/cluster2-client.mirror.70755.log.gz 2024-05-08T01:02:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40689.log 2024-05-08T01:02:08.796 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82030.log 2024-05-08T01:02:08.796 INFO:teuthology.orchestra.run.smithi116.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89959.log.gz 2024-05-08T01:02:08.797 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40689.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93887.log.gz 2024-05-08T01:02:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.94647.log 2024-05-08T01:02:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68574.log 2024-05-08T01:02:08.799 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.94647.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.94647.log.gz 2024-05-08T01:02:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93374.log 2024-05-08T01:02:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93062.log 2024-05-08T01:02:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93374.log: /var/log/ceph/cluster2-client.mirror.68574.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68574.log.gz 2024-05-08T01:02:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36734.log 2024-05-08T01:02:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93062.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93062.log.gz 2024-05-08T01:02:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73683.log 2024-05-08T01:02:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.36734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77523.log 91.0% -- replaced with /var/log/ceph/cluster2-client.mirror.93374.log.gz 2024-05-08T01:02:08.806 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.806 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.73683.log: -- replaced with /var/log/ceph/cluster2-client.mirror.40689.log.gz 2024-05-08T01:02:08.807 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73683.log.gz 2024-05-08T01:02:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47975.log 2024-05-08T01:02:08.808 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.77523.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48053.log.gz 2024-05-08T01:02:08.808 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77523.log.gz 2024-05-08T01:02:08.811 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90926.log 2024-05-08T01:02:08.811 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52566.log.gz 2024-05-08T01:02:08.811 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72961.log 2024-05-08T01:02:08.811 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47975.log: /var/log/ceph/cluster1-client.mirror.90926.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90926.log.gz 2024-05-08T01:02:08.811 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55916.log 2024-05-08T01:02:08.816 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72961.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49754.log.gz 2024-05-08T01:02:08.829 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53193.log.gz 2024-05-08T01:02:08.832 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.55916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65643.log 2024-05-08T01:02:08.833 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87670.log 2024-05-08T01:02:08.834 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.65643.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65643.log.gz 2024-05-08T01:02:08.835 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47975.log.gz 2024-05-08T01:02:08.837 INFO:teuthology.orchestra.run.smithi116.stderr: 54.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36734.log.gz 2024-05-08T01:02:08.839 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68728.log 2024-05-08T01:02:08.840 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87670.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87670.log.gz 2024-05-08T01:02:08.840 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93956.log 2024-05-08T01:02:08.842 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.68728.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.68728.log.gz 2024-05-08T01:02:08.843 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84803.log 2024-05-08T01:02:08.844 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.93956.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.93956.log.gz 2024-05-08T01:02:08.844 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41848.log 2024-05-08T01:02:08.846 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84803.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.84803.log.gz 2024-05-08T01:02:08.848 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89284.log 2024-05-08T01:02:08.848 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69605.log 2024-05-08T01:02:08.848 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89284.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89284.log.gz 2024-05-08T01:02:08.849 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51466.log 2024-05-08T01:02:08.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69605.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69605.log.gz 2024-05-08T01:02:08.851 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90769.log 2024-05-08T01:02:08.852 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90004.log 2024-05-08T01:02:08.852 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90769.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90769.log.gz 2024-05-08T01:02:08.854 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92298.log 2024-05-08T01:02:08.854 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90004.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90004.log.gz 2024-05-08T01:02:08.854 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84832.log 2024-05-08T01:02:08.855 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92298.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92298.log.gz 2024-05-08T01:02:08.856 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36108.log 2024-05-08T01:02:08.856 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84832.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84832.log.gz 2024-05-08T01:02:08.859 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36580.log 2024-05-08T01:02:08.859 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36108.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36108.log.gz 2024-05-08T01:02:08.859 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85506.log 2024-05-08T01:02:08.860 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.36580.log: gzip 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51466.log.gz 2024-05-08T01:02:08.860 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53455.log 2024-05-08T01:02:08.861 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85506.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85506.log.gz 2024-05-08T01:02:08.862 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35604.log 2024-05-08T01:02:08.863 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.53455.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53455.log.gz 2024-05-08T01:02:08.863 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89914.log 2024-05-08T01:02:08.863 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.35604.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35604.log.gz 2024-05-08T01:02:08.864 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88734.log 2024-05-08T01:02:08.864 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89914.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89914.log.gz 2024-05-08T01:02:08.866 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 56.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36580.log.gz 2024-05-08T01:02:08.866 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85875.log 2024-05-08T01:02:08.866 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88734.log: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60287.log.gz 2024-05-08T01:02:08.866 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88734.log.gz 2024-05-08T01:02:08.866 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50759.log 2024-05-08T01:02:08.867 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44504.log 2024-05-08T01:02:08.869 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50759.log: /var/log/ceph/cluster2-client.mirror.85875.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85875.log.gz 2024-05-08T01:02:08.869 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55623.log 2024-05-08T01:02:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79217.log 2024-05-08T01:02:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.55623.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55623.log.gz 2024-05-08T01:02:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37091.log 2024-05-08T01:02:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50426.log 2024-05-08T01:02:08.873 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37091.log: /var/log/ceph/cluster2-client.mirror.79217.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79217.log.gz 2024-05-08T01:02:08.873 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.36916.log 2024-05-08T01:02:08.875 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72919.log 2024-05-08T01:02:08.875 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.1.36916.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.39256.log.gz 2024-05-08T01:02:08.876 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64526.log 2024-05-08T01:02:08.876 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83556.log 2024-05-08T01:02:08.878 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.64526.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72919.log.gz 2024-05-08T01:02:08.878 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41848.log.gz 2024-05-08T01:02:08.878 INFO:teuthology.orchestra.run.smithi116.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64526.log.gz 2024-05-08T01:02:08.880 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46538.log 2024-05-08T01:02:08.881 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54778.log 2024-05-08T01:02:08.881 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83556.log.gz 2024-05-08T01:02:08.881 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58917.log 2024-05-08T01:02:08.881 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.54778.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51951.log.gz 2024-05-08T01:02:08.889 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46910.log 2024-05-08T01:02:08.896 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.58917.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44504.log.gz 2024-05-08T01:02:08.897 INFO:teuthology.orchestra.run.smithi116.stderr: 19.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.36916.log.gz 2024-05-08T01:02:08.897 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37091.log.gz 2024-05-08T01:02:08.900 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90634.log 2024-05-08T01:02:08.901 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46910.log: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.37068.log.gz 2024-05-08T01:02:08.902 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83699.log 2024-05-08T01:02:08.902 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90634.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90634.log.gz 2024-05-08T01:02:08.904 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45833.log 2024-05-08T01:02:08.904 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83699.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83699.log.gz 2024-05-08T01:02:08.904 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49819.log 2024-05-08T01:02:08.904 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54755.log 2024-05-08T01:02:08.905 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90566.log 2024-05-08T01:02:08.906 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.54755.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54755.log.gz 2024-05-08T01:02:08.908 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41530.log 2024-05-08T01:02:08.908 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90566.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90566.log.gz 2024-05-08T01:02:08.908 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64812.log 2024-05-08T01:02:08.909 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86859.log 2024-05-08T01:02:08.909 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.64812.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64812.log.gz 2024-05-08T01:02:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64879.log 2024-05-08T01:02:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86859.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56825.log.gz 2024-05-08T01:02:08.912 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86859.log.gz 2024-05-08T01:02:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92236.log 2024-05-08T01:02:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.64879.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64879.log.gz 2024-05-08T01:02:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72541.log 2024-05-08T01:02:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92236.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.92236.log.gz 2024-05-08T01:02:08.914 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90251.log 2024-05-08T01:02:08.914 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95037.log 2024-05-08T01:02:08.914 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90251.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90251.log.gz 2024-05-08T01:02:08.914 INFO:teuthology.orchestra.run.smithi116.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72541.log.gz 2024-05-08T01:02:08.915 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.95037.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.95037.log.gz 2024-05-08T01:02:08.916 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49426.log 2024-05-08T01:02:08.918 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48033.log 2024-05-08T01:02:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40317.log 2024-05-08T01:02:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48033.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85263.log 2024-05-08T01:02:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40317.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51006.log 2024-05-08T01:02:08.919 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85263.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85263.log.gz 2024-05-08T01:02:08.929 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40317.log.gz 2024-05-08T01:02:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84423.log 2024-05-08T01:02:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75950.log 2024-05-08T01:02:08.932 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84423.log: /var/log/ceph/cluster2-client.mirror.51006.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84423.log.gz 2024-05-08T01:02:08.933 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41530.log.gz 2024-05-08T01:02:08.937 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72961.log.gz 2024-05-08T01:02:08.940 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47704.log.gz 2024-05-08T01:02:08.947 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86508.log 2024-05-08T01:02:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.75950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93820.log 2024-05-08T01:02:08.948 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82008.log 2024-05-08T01:02:08.949 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93820.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93820.log.gz 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.86508.log.gz 2024-05-08T01:02:08.950 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91922.log 2024-05-08T01:02:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.82008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41999.log 2024-05-08T01:02:08.950 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91922.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.91922.log.gz 2024-05-08T01:02:08.953 INFO:teuthology.orchestra.run.smithi116.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.82008.log.gz 2024-05-08T01:02:08.953 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87239.log 2024-05-08T01:02:08.953 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41999.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46538.log.gz 2024-05-08T01:02:08.953 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79603.log 2024-05-08T01:02:08.953 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75950.log.gz 2024-05-08T01:02:08.953 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46910.log.gz 2024-05-08T01:02:08.953 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74069.log 2024-05-08T01:02:08.954 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79603.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87239.log.gz 2024-05-08T01:02:08.954 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49426.log.gz 2024-05-08T01:02:08.954 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79603.log.gz 2024-05-08T01:02:08.956 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50759.log.gz 2024-05-08T01:02:08.960 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88543.log 2024-05-08T01:02:08.961 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41809.log 2024-05-08T01:02:08.962 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88543.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.88543.log.gz 2024-05-08T01:02:08.962 INFO:teuthology.orchestra.run.smithi116.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74069.log.gz 2024-05-08T01:02:08.970 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35436.log 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90701.log 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35436.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35436.log.gz 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37423.log 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90701.log: gzip 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.90701.log.gz 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88101.log 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.37423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35730.log 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88101.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85528.log 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35730.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35730.log.gz 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88101.log.gz 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92767.log 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75918.log 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85528.log.gz/var/log/ceph/cluster2-client.mirror.92767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76603.log 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75918.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75918.log.gz 2024-05-08T01:02:08.971 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41809.log.gz 2024-05-08T01:02:08.972 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:08.972 INFO:teuthology.orchestra.run.smithi116.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92767.log.gz 2024-05-08T01:02:08.972 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52499.log 2024-05-08T01:02:08.972 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41999.log.gz 2024-05-08T01:02:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.76603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47831.log 2024-05-08T01:02:08.973 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90026.log 2024-05-08T01:02:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.47831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51661.log 2024-05-08T01:02:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90026.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90026.log.gz 2024-05-08T01:02:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87832.log 2024-05-08T01:02:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.51661.log: -- /var/log/ceph/cluster2-client.mirror.50207.log 2024-05-08T01:02:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49730.log 2024-05-08T01:02:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50207.log: /var/log/ceph/cluster2-client.mirror.87832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61526.log 2024-05-08T01:02:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90161.log 2024-05-08T01:02:08.980 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.61526.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87832.log.gz 2024-05-08T01:02:08.980 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77909.log 2024-05-08T01:02:08.981 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90161.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90161.log.gz 2024-05-08T01:02:08.981 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72939.log 2024-05-08T01:02:08.983 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.77909.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77909.log.gz 2024-05-08T01:02:08.983 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42886.log 2024-05-08T01:02:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.72939.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72939.log.gz 2024-05-08T01:02:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87188.log 2024-05-08T01:02:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.42886.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43198.log 2024-05-08T01:02:08.990 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87188.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87188.log.gz 2024-05-08T01:02:08.992 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73325.log 2024-05-08T01:02:08.992 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47507.log 2024-05-08T01:02:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.73325.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73325.log.gz 2024-05-08T01:02:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64434.log 2024-05-08T01:02:08.995 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64182.log 2024-05-08T01:02:08.996 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.64434.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64434.log.gz 2024-05-08T01:02:08.997 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36657.log 2024-05-08T01:02:08.999 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.64182.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45833.log.gz 2024-05-08T01:02:08.999 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47507.log.gz 2024-05-08T01:02:09.013 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49777.log 2024-05-08T01:02:09.013 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36657.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36657.log.gz 2024-05-08T01:02:09.014 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48508.log 2024-05-08T01:02:09.017 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49777.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.47831.log.gz 2024-05-08T01:02:09.018 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43198.log.gz 2024-05-08T01:02:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69233.log 2024-05-08T01:02:09.021 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48508.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51661.log.gz 2024-05-08T01:02:09.022 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48463.log.gz 2024-05-08T01:02:09.031 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53025.log.gz 2024-05-08T01:02:09.034 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89621.log 2024-05-08T01:02:09.049 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69233.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38744.log.gz 2024-05-08T01:02:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39287.log 2024-05-08T01:02:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89621.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89621.log.gz 2024-05-08T01:02:09.052 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76603.log.gz 2024-05-08T01:02:09.052 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34722.log 2024-05-08T01:02:09.052 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.39287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69002.log 2024-05-08T01:02:09.053 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34722.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34722.log.gz 2024-05-08T01:02:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69002.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69002.log.gz 2024-05-08T01:02:09.057 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83966.log 2024-05-08T01:02:09.057 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42220.log 2024-05-08T01:02:09.058 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.83966.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.83966.log.gz 2024-05-08T01:02:09.058 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86908.log 2024-05-08T01:02:09.059 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37392.log 2024-05-08T01:02:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86908.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86908.log.gz 2024-05-08T01:02:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87001.log 2024-05-08T01:02:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.37392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74667.log 2024-05-08T01:02:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87001.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87001.log.gz 2024-05-08T01:02:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90184.log 2024-05-08T01:02:09.070 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.74667.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74667.log.gz 2024-05-08T01:02:09.070 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85755.log 2024-05-08T01:02:09.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90184.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90184.log.gz 2024-05-08T01:02:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71796.log 2024-05-08T01:02:09.080 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.39287.log.gz 2024-05-08T01:02:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85755.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42220.log.gz 2024-05-08T01:02:09.082 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85755.log.gz 2024-05-08T01:02:09.082 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50873.log 2024-05-08T01:02:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.71796.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71796.log.gz 2024-05-08T01:02:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80403.log 2024-05-08T01:02:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56219.log 2024-05-08T01:02:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80403.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80403.log.gz 2024-05-08T01:02:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92882.log 2024-05-08T01:02:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.56219.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56219.log.gz 2024-05-08T01:02:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71519.log 2024-05-08T01:02:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92882.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92882.log.gz 2024-05-08T01:02:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51601.log 2024-05-08T01:02:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.71519.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71519.log.gz 2024-05-08T01:02:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72827.log 2024-05-08T01:02:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.51601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37631.log 2024-05-08T01:02:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72827.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72827.log.gz 2024-05-08T01:02:09.095 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89149.log 2024-05-08T01:02:09.095 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.37631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49561.log 2024-05-08T01:02:09.098 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89149.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89149.log.gz 2024-05-08T01:02:09.099 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44039.log 2024-05-08T01:02:09.104 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64182.log.gz 2024-05-08T01:02:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67794.log 2024-05-08T01:02:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34666.log 2024-05-08T01:02:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.67794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42185.log 2024-05-08T01:02:09.104 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34666.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34666.log.gz 2024-05-08T01:02:09.105 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51872.log 2024-05-08T01:02:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42185.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67794.log.gz 2024-05-08T01:02:09.106 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88294.log 2024-05-08T01:02:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51872.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85406.log 2024-05-08T01:02:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89801.log 2024-05-08T01:02:09.108 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88294.log.gz 2024-05-08T01:02:09.108 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85406.log: /var/log/ceph/cluster1-client.mirror.89801.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.89801.log.gz 2024-05-08T01:02:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61744.log 2024-05-08T01:02:09.110 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.37392.log.gz 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.37423.log.gz 2024-05-08T01:02:09.110 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:09.110 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85406.log.gz 2024-05-08T01:02:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45979.log 2024-05-08T01:02:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.61744.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61744.log.gz 2024-05-08T01:02:09.111 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49561.log.gz 2024-05-08T01:02:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91129.log 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50649.log 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91129.log: gzip -5 --verbose -- 86.7% /var/log/ceph/cluster1-client.mirror.90544.log 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.91129.log.gz 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37451.log 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50649.log: /var/log/ceph/cluster1-client.mirror.90544.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90544.log.gz 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88243.log 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72516.log 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88243.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92970.log 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.72516.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72516.log.gz 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88243.log.gz 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89824.log 2024-05-08T01:02:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89216.log 2024-05-08T01:02:09.121 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89824.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89824.log.gz 2024-05-08T01:02:09.121 INFO:teuthology.orchestra.run.smithi116.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49730.log.gz 2024-05-08T01:02:09.122 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39127.log 2024-05-08T01:02:09.123 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89216.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89216.log.gz 2024-05-08T01:02:09.125 INFO:teuthology.orchestra.run.smithi116.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50383.log.gz 2024-05-08T01:02:09.136 INFO:teuthology.orchestra.run.smithi116.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51006.log.gz 2024-05-08T01:02:09.137 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42278.log 2024-05-08T01:02:09.138 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.39127.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51601.log.gz 2024-05-08T01:02:09.139 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42185.log.gz 2024-05-08T01:02:09.139 INFO:teuthology.orchestra.run.smithi116.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51872.log.gz 2024-05-08T01:02:09.145 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42886.log.gz 2024-05-08T01:02:09.146 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89734.log 2024-05-08T01:02:09.147 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48396.log.gz 2024-05-08T01:02:09.149 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83269.log 2024-05-08T01:02:09.149 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89734.log: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89734.log.gz 2024-05-08T01:02:09.149 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50825.log.gz 2024-05-08T01:02:09.150 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50626.log 2024-05-08T01:02:09.155 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83269.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83269.log.gz 2024-05-08T01:02:09.155 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90814.log 2024-05-08T01:02:09.157 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65302.log 2024-05-08T01:02:09.157 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90814.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.90814.log.gz 2024-05-08T01:02:09.158 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69570.log 2024-05-08T01:02:09.160 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.65302.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65302.log.gz 2024-05-08T01:02:09.160 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82341.log 2024-05-08T01:02:09.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69570.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69570.log.gz 2024-05-08T01:02:09.169 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54778.log.gz 2024-05-08T01:02:09.169 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55916.log.gz 2024-05-08T01:02:09.169 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/cluster2-client.mirror.85292.log 2024-05-08T01:02:09.169 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42650.log 2024-05-08T01:02:09.169 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.82341.log: gzip -5/var/log/ceph/cluster2-client.mirror.85292.log: --verbose -- /var/log/ceph/cluster1-client.mirror.90859.log 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82341.log.gz 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42650.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85292.log.gz 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88772.log 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.90859.log: /var/log/ceph/cluster2-client.mirror.58167.log 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88772.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90859.log.gz 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88772.log.gz 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37068.log 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.58167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88152.log 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.37068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90049.log 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88152.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44039.log.gz 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43760.log 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88152.log.gz 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90049.log: 86.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91500.log 2024-05-08T01:02:09.170 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.90049.log.gz 2024-05-08T01:02:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44787.log 2024-05-08T01:02:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.91500.log: /var/log/ceph/cluster1-client.mirror.89306.log 2024-05-08T01:02:09.171 INFO:teuthology.orchestra.run.smithi116.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91500.log.gz 2024-05-08T01:02:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39980.log 2024-05-08T01:02:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70123.log 2024-05-08T01:02:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39980.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89306.log.gz 2024-05-08T01:02:09.174 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68180.log 2024-05-08T01:02:09.174 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.70123.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70123.log.gz 2024-05-08T01:02:09.174 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87932.log 2024-05-08T01:02:09.174 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.68180.log: /var/log/ceph/cluster2-client.mirror.47859.log 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68180.log.gz 2024-05-08T01:02:09.176 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87932.log: 2024-05-08T01:02:09.176 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87932.log.gz 2024-05-08T01:02:09.176 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76647.log 2024-05-08T01:02:09.176 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.37015.log 2024-05-08T01:02:09.176 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.76647.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76647.log.gz 2024-05-08T01:02:09.178 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.1.37015.log: /var/log/ceph/cluster2-client.mirror.47859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78766.log 2024-05-08T01:02:09.178 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89981.log 2024-05-08T01:02:09.179 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.78766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81224.log 2024-05-08T01:02:09.179 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89981.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78766.log.gz 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89981.log.gz 2024-05-08T01:02:09.179 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:09.184 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81224.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81224.log.gz 2024-05-08T01:02:09.184 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79237.log 2024-05-08T01:02:09.184 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72449.log 2024-05-08T01:02:09.186 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.79237.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.79237.log.gz 2024-05-08T01:02:09.186 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51574.log 2024-05-08T01:02:09.186 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62444.log 2024-05-08T01:02:09.187 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.72449.log: /var/log/ceph/cluster1-client.mirror.51574.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72449.log.gz 2024-05-08T01:02:09.189 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.62444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41472.log 2024-05-08T01:02:09.190 INFO:teuthology.orchestra.run.smithi116.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50426.log.gz 2024-05-08T01:02:09.191 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45345.log 2024-05-08T01:02:09.191 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.36951.log 2024-05-08T01:02:09.192 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64501.log 2024-05-08T01:02:09.195 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.0.36951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85243.log 2024-05-08T01:02:09.198 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.64501.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64501.log.gz 2024-05-08T01:02:09.198 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92559.log 2024-05-08T01:02:09.198 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85243.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85243.log.gz 2024-05-08T01:02:09.198 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69211.log 2024-05-08T01:02:09.200 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92559.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92559.log.gz 2024-05-08T01:02:09.200 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64847.log 2024-05-08T01:02:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.69211.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69211.log.gz 2024-05-08T01:02:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48543.log 2024-05-08T01:02:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.64847.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82030.log.gz 2024-05-08T01:02:09.205 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64847.log.gz 2024-05-08T01:02:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45953.log 2024-05-08T01:02:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69904.log 2024-05-08T01:02:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.45953.log: /var/log/ceph/cluster1-client.admin.48543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64138.log 2024-05-08T01:02:09.208 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51049.log 2024-05-08T01:02:09.208 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.64138.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64138.log.gz 2024-05-08T01:02:09.208 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90476.log 2024-05-08T01:02:09.209 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51049.log: /var/log/ceph/cluster1-client.mirror.90476.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90476.log.gz 2024-05-08T01:02:09.212 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34806.log 2024-05-08T01:02:09.212 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58894.log 2024-05-08T01:02:09.213 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34806.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34806.log.gz 2024-05-08T01:02:09.218 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74012.log 2024-05-08T01:02:09.218 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.58894.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58894.log.gz 2024-05-08T01:02:09.218 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40596.log 2024-05-08T01:02:09.223 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74012.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74012.log.gz 2024-05-08T01:02:09.223 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58917.log.gz 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44822.log 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71410.log 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86579.log 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.71410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88634.log 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86579.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86579.log.gz 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51574.log.gz 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90319.log 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.48543.log.gz 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88634.log: 20.5% 20.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.36951.log.gz 2024-05-08T01:02:09.224 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.37015.log.gz 2024-05-08T01:02:09.225 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88634.log.gz 2024-05-08T01:02:09.225 INFO:teuthology.orchestra.run.smithi116.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71410.log.gz 2024-05-08T01:02:09.226 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39980.log.gz 2024-05-08T01:02:09.293 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40596.log.gz 2024-05-08T01:02:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49030.log 2024-05-08T01:02:09.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90319.log: gzip -5 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.90319.log.gz 2024-05-08T01:02:09.293 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.40914.log 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46945.log 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38043.log 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87359.log 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68683.log 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87359.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87359.log.gz 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.68683.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68683.log.gz 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90206.log 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93035.log 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90206.log: gzip -5 --verbose -- 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.90206.log.gz 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/cluster1-client.mirror.72130.log 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/cluster1-client.mirror.93035.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90521.log 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93035.log.gz 2024-05-08T01:02:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.72130.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69904.log.gz 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35184.log 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72130.log.gz 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90521.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40166.log 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90521.log.gz/var/log/ceph/cluster2-client.admin.35184.log: 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35184.log.gz 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38043.log.gz 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72155.log 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89441.log 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43760.log.gz 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.72155.log: -- /var/log/ceph/cluster2-client.mirror.50340.log 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89441.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89441.log.gz 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92944.log 2024-05-08T01:02:09.295 INFO:teuthology.orchestra.run.smithi116.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72155.log.gzgzip 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50802.log 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50340.log: /var/log/ceph/cluster2-client.mirror.92944.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41344.log 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51356.log 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92944.log.gz 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49030.log.gz 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87021.log 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45252.log 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42499.log 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94096.log 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.42499.log: -- /var/log/ceph/cluster2-client.mirror.40073.log 2024-05-08T01:02:09.296 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87021.log.gz 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94454.log 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94096.log: /var/log/ceph/cluster2-client.mirror.40073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88059.log 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.94454.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94454.log.gz 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80435.log 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88059.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88059.log.gz 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50250.log 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.80435.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94096.log.gz 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80435.log.gz 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.91376.log 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94401.log 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.0.91376.log: /var/log/ceph/cluster2-client.mirror.50250.log: 20.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.91376.log.gz 2024-05-08T01:02:09.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88401.log 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86979.log 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49596.log 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88401.log.gz 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86979.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86979.log.gz 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89005.log 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94401.log.gz 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86366.log 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.89005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83633.log 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76581.log 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.83633.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86366.log.gz 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90724.log 2024-05-08T01:02:09.298 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.89005.log.gz 2024-05-08T01:02:09.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.76581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72862.log 2024-05-08T01:02:09.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90724.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.83633.log.gz 2024-05-08T01:02:09.299 INFO:teuthology.orchestra.run.smithi116.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.90724.log.gz 2024-05-08T01:02:09.299 INFO:teuthology.orchestra.run.smithi116.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.76581.log.gz 2024-05-08T01:02:09.299 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86295.log 2024-05-08T01:02:09.299 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65954.log 2024-05-08T01:02:09.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86295.log: /var/log/ceph/cluster2-client.mirror.72862.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72862.log.gz 2024-05-08T01:02:09.299 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86295.log.gz 2024-05-08T01:02:09.299 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37950.log 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65910.log 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37950.log: /var/log/ceph/cluster2-client.mirror.65954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46201.log 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.65910.log: -- replaced with /var/log/ceph/cluster2-client.mirror.65954.log.gz 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44601.log 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75211.log 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37352.log 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83588.log 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86479.log 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.83588.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75211.log.gz 2024-05-08T01:02:09.300 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83588.log.gz 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36836.log 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87719.log 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.36836.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.86479.log.gz 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61526.log.gz 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40131.log 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87719.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95331.log 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40131.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87719.log.gz 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37985.log 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.95331.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.95331.log.gz 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82757.log 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76714.log 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47724.log 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82757.log.gz 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.76714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74289.log 2024-05-08T01:02:09.301 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76714.log.gz 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.47724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65989.log 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47632.log 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.74289.log: /var/log/ceph/cluster2-client.mirror.65989.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65989.log.gz 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74289.log.gz 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42278.log.gz 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88519.log 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42650.log.gz 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64469.log 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.47632.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88272.log 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.64469.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88519.log.gz 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93799.log 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64469.log.gz 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86717.log 2024-05-08T01:02:09.302 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88272.log: /var/log/ceph/cluster2-client.mirror.93799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71484.log 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86717.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86717.log.gz 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88272.log.gz 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48767.log 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.71484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89576.log 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71484.log.gz 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41472.log.gz 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48767.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93799.log.gz 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64546.log 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36614.log 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89576.log: /var/log/ceph/cluster1-client.mirror.64546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69099.log 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.36614.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89576.log.gz 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34971.log 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47677.log 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69099.log.gz 2024-05-08T01:02:09.303 INFO:teuthology.orchestra.run.smithi116.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64546.log.gz 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89689.log 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34971.log.gz 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.47677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40259.log 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91350.log 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89689.log: /var/log/ceph/cluster2-client.mirror.40259.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89689.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35520.log 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91350.log: 55.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91350.log.gz 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62006.log 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36911.log 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.36911.log: /var/log/ceph/cluster2-client.mirror.62006.log: /var/log/ceph/cluster2-client.admin.35520.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35520.log.gz 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48236.log 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49638.log 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62875.log 2024-05-08T01:02:09.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93779.log 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.62875.log: /var/log/ceph/cluster2-client.mirror.49638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36802.log 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr: 1.3%/var/log/ceph/cluster2-client.mirror.93779.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62875.log.gz 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.93779.log.gz 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91106.log 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.36802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46002.log 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84445.log 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91106.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91106.log.gz 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55008.log 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57832.log 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.84445.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58167.log.gz 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.55008.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84445.log.gz 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65668.log 2024-05-08T01:02:09.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.57832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35224.log 2024-05-08T01:02:09.307 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.65668.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65668.log.gz 2024-05-08T01:02:09.307 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91446.log 2024-05-08T01:02:09.308 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35224.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35224.log.gz 2024-05-08T01:02:09.308 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75540.log 2024-05-08T01:02:09.314 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91446.log: 88.4% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44787.log.gz 2024-05-08T01:02:09.314 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44822.log.gz 2024-05-08T01:02:09.314 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46724.log 2024-05-08T01:02:09.314 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75540.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75540.log.gz 2024-05-08T01:02:09.318 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45252.log.gz 2024-05-08T01:02:09.320 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45345.log.gz 2024-05-08T01:02:09.322 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46945.log.gz 2024-05-08T01:02:09.327 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47632.log.gz 2024-05-08T01:02:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43009.log 2024-05-08T01:02:09.337 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46724.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50873.log.gz 2024-05-08T01:02:09.342 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44601.log.gz 2024-05-08T01:02:09.343 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41344.log.gz 2024-05-08T01:02:09.343 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49638.log.gz 2024-05-08T01:02:09.343 INFO:teuthology.orchestra.run.smithi116.stderr: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36614.log.gz 2024-05-08T01:02:09.345 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40914.log.gz 2024-05-08T01:02:09.345 INFO:teuthology.orchestra.run.smithi116.stderr: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36802.log.gz 2024-05-08T01:02:09.345 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51356.log.gz 2024-05-08T01:02:09.346 INFO:teuthology.orchestra.run.smithi116.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36836.log.gz 2024-05-08T01:02:09.347 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40073.log.gz 2024-05-08T01:02:09.347 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40131.log.gz 2024-05-08T01:02:09.347 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40259.log.gz 2024-05-08T01:02:09.347 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40166.log.gz 2024-05-08T01:02:09.348 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87812.log 2024-05-08T01:02:09.348 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42499.log.gz 2024-05-08T01:02:09.348 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.43009.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37950.log.gz 2024-05-08T01:02:09.349 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37985.log.gz 2024-05-08T01:02:09.351 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48767.log.gz 2024-05-08T01:02:09.359 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81731.log 2024-05-08T01:02:09.359 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87812.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87812.log.gz 2024-05-08T01:02:09.377 INFO:teuthology.orchestra.run.smithi116.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49777.log.gz 2024-05-08T01:02:09.385 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46201.log.gz 2024-05-08T01:02:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76304.log 2024-05-08T01:02:09.386 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81731.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81731.log.gz 2024-05-08T01:02:09.395 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41623.log 2024-05-08T01:02:09.396 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.76304.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76304.log.gz 2024-05-08T01:02:09.398 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46724.log.gz 2024-05-08T01:02:09.399 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37631.log.gz 2024-05-08T01:02:09.399 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49596.log.gz 2024-05-08T01:02:09.402 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48033.log.gz 2024-05-08T01:02:09.410 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48122.log 2024-05-08T01:02:09.413 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41623.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50207.log.gz 2024-05-08T01:02:09.418 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69233.log.gz 2024-05-08T01:02:09.420 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79546.log 2024-05-08T01:02:09.420 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50626.log.gz 2024-05-08T01:02:09.421 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92005.log 2024-05-08T01:02:09.422 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79546.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79546.log.gz 2024-05-08T01:02:09.422 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51092.log 2024-05-08T01:02:09.422 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92005.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.92005.log.gz 2024-05-08T01:02:09.424 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92742.log 2024-05-08T01:02:09.424 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36631.log 2024-05-08T01:02:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92742.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92742.log.gz 2024-05-08T01:02:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78788.log 2024-05-08T01:02:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36631.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36631.log.gz 2024-05-08T01:02:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40631.log 2024-05-08T01:02:09.432 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.78788.log: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78788.log.gz 2024-05-08T01:02:09.432 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35096.log 2024-05-08T01:02:09.435 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37653.log 2024-05-08T01:02:09.435 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35096.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35096.log.gz 2024-05-08T01:02:09.435 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73213.log 2024-05-08T01:02:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37653.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50649.log.gz 2024-05-08T01:02:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79982.log 2024-05-08T01:02:09.437 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.73213.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73213.log.gz 2024-05-08T01:02:09.437 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40631.log.gz 2024-05-08T01:02:09.438 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91333.log 2024-05-08T01:02:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79982.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79982.log.gz 2024-05-08T01:02:09.443 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41623.log.gz 2024-05-08T01:02:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34848.log 2024-05-08T01:02:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91333.log: 54.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91333.log.gz 2024-05-08T01:02:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46387.log 2024-05-08T01:02:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.34848.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34848.log.gz 2024-05-08T01:02:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75231.log 2024-05-08T01:02:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90454.log 2024-05-08T01:02:09.445 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.75231.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75231.log.gz 2024-05-08T01:02:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47746.log 2024-05-08T01:02:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90454.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90454.log.gz 2024-05-08T01:02:09.449 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91196.log 2024-05-08T01:02:09.449 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90611.log 2024-05-08T01:02:09.449 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91196.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.91196.log.gz 2024-05-08T01:02:09.450 INFO:teuthology.orchestra.run.smithi116.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49819.log.gz 2024-05-08T01:02:09.451 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38656.log 2024-05-08T01:02:09.451 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90611.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90611.log.gz 2024-05-08T01:02:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-05-08T01:02:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.38656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42843.log 2024-05-08T01:02:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2.log: 87.0% -- replaced with /var/log/ceph/cluster2.log.gz 2024-05-08T01:02:09.459 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43667.log 2024-05-08T01:02:09.460 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.42843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42727.log 2024-05-08T01:02:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.43667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56078.log 2024-05-08T01:02:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74622.log 2024-05-08T01:02:09.478 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.56078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45101.log 2024-05-08T01:02:09.482 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.74622.log: 88.1% 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65910.log.gz 2024-05-08T01:02:09.484 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47228.log 2024-05-08T01:02:09.485 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.45101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92715.log 2024-05-08T01:02:09.486 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.47228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94504.log 2024-05-08T01:02:09.487 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92715.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.92715.log.gz 2024-05-08T01:02:09.490 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82695.log 2024-05-08T01:02:09.491 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.74622.log.gz 2024-05-08T01:02:09.492 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.94504.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.94504.log.gz 2024-05-08T01:02:09.493 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77100.log 2024-05-08T01:02:09.494 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82695.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82695.log.gz 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75154.log 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.77100.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45101.log.gz 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77100.log.gz 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46387.log.gz 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42557.log 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.75154.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47228.log.gz 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75154.log.gz 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93202.log 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49484.log 2024-05-08T01:02:09.501 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5%/var/log/ceph/cluster2-client.mirror.93202.log: -- replaced with /var/log/ceph/cluster2-client.mirror.51092.log.gz 2024-05-08T01:02:09.502 INFO:teuthology.orchestra.run.smithi116.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93202.log.gz 2024-05-08T01:02:09.507 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95121.log 2024-05-08T01:02:09.507 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44539.log 2024-05-08T01:02:09.507 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.95121.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.95121.log.gz 2024-05-08T01:02:09.507 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93861.log 2024-05-08T01:02:09.507 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44446.log 2024-05-08T01:02:09.508 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93861.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93861.log.gz 2024-05-08T01:02:09.508 INFO:teuthology.orchestra.run.smithi116.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62006.log.gz 2024-05-08T01:02:09.509 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38872.log 2024-05-08T01:02:09.510 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44446.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36785.log 2024-05-08T01:02:09.513 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.38872.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52061.log 2024-05-08T01:02:09.514 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.36785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.94995.log 2024-05-08T01:02:09.514 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41906.log 2024-05-08T01:02:09.514 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.94995.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.94995.log.gz 2024-05-08T01:02:09.514 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59104.log 2024-05-08T01:02:09.514 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47100.log 2024-05-08T01:02:09.514 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.59104.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59104.log.gz 2024-05-08T01:02:09.514 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90949.log 2024-05-08T01:02:09.516 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66632.log 2024-05-08T01:02:09.516 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90949.log: /var/log/ceph/cluster2-client.mirror.47100.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.90949.log.gz 2024-05-08T01:02:09.516 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93591.log 2024-05-08T01:02:09.517 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.66632.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45953.log.gz 2024-05-08T01:02:09.517 INFO:teuthology.orchestra.run.smithi116.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66632.log.gz 2024-05-08T01:02:09.518 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69191.log 2024-05-08T01:02:09.519 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51310.log 2024-05-08T01:02:09.521 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.69191.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69191.log.gz 2024-05-08T01:02:09.523 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87883.log 2024-05-08T01:02:09.525 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66540.log 2024-05-08T01:02:09.525 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87883.log: 88.1% 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50250.log.gz 2024-05-08T01:02:09.525 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.87883.log.gz 2024-05-08T01:02:09.525 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-05-08T01:02:09.528 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.66540.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66540.log.gz 2024-05-08T01:02:09.528 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92215.log 2024-05-08T01:02:09.528 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88592.log 2024-05-08T01:02:09.528 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92215.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.92215.log.gz 2024-05-08T01:02:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76336.log 2024-05-08T01:02:09.532 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88592.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88592.log.gz 2024-05-08T01:02:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91588.log 2024-05-08T01:02:09.536 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.76336.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76336.log.gz 2024-05-08T01:02:09.536 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42931.log 2024-05-08T01:02:09.536 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91588.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91588.log.gz 2024-05-08T01:02:09.544 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83734.log 2024-05-08T01:02:09.548 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38171.log 2024-05-08T01:02:09.548 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83734.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.83734.log.gz 2024-05-08T01:02:09.548 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44446.log.gz 2024-05-08T01:02:09.550 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49122.log 2024-05-08T01:02:09.552 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73305.log 2024-05-08T01:02:09.552 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79511.log 2024-05-08T01:02:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.73305.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73305.log.gz 2024-05-08T01:02:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89396.log 2024-05-08T01:02:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79511.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92970.log.gz 2024-05-08T01:02:09.557 INFO:teuthology.orchestra.run.smithi116.stderr: 64.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36785.log.gz 2024-05-08T01:02:09.557 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.79511.log.gz 2024-05-08T01:02:09.563 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41906.log.gz 2024-05-08T01:02:09.563 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.38872.log.gz 2024-05-08T01:02:09.563 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42727.log.gz 2024-05-08T01:02:09.563 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42557.log.gz 2024-05-08T01:02:09.563 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81602.log 2024-05-08T01:02:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89396.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89396.log.gz 2024-05-08T01:02:09.564 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51138.log 2024-05-08T01:02:09.564 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38171.log.gz 2024-05-08T01:02:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44411.log 2024-05-08T01:02:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50845.log 2024-05-08T01:02:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44411.log: 88.7% 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81602.log.gz 2024-05-08T01:02:09.564 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43667.log.gz 2024-05-08T01:02:09.564 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42406.log 2024-05-08T01:02:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81132.log 2024-05-08T01:02:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50403.log 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81132.log: gzip/var/log/ceph/cluster1-client.mirror.50403.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81132.log.gz 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57809.log 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40445.log 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.57809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91552.log 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57809.log.gz 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.40445.log: --verbose -- /var/log/ceph/cluster2-client.admin.35098.log 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36495.log 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.35098.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35098.log.gz 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88010.log 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91552.log.gz 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.36495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71432.log 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71365.log 2024-05-08T01:02:09.576 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.71432.log: /var/log/ceph/cluster2-client.mirror.88010.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.88010.log.gz 2024-05-08T01:02:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26858.log 2024-05-08T01:02:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.71365.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71365.log.gz 2024-05-08T01:02:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48602.log 2024-05-08T01:02:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.tmp-client.admin.26858.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26858.log.gz 2024-05-08T01:02:09.577 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36150.log 2024-05-08T01:02:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.48602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42313.log 2024-05-08T01:02:09.579 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.36150.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36150.log.gz 2024-05-08T01:02:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57642.log 2024-05-08T01:02:09.580 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.42313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86837.log 2024-05-08T01:02:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.57642.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50340.log.gz 2024-05-08T01:02:09.588 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38656.log.gz 2024-05-08T01:02:09.589 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.43009.log.gz 2024-05-08T01:02:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52244.log 2024-05-08T01:02:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.86837.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.86837.log.gz 2024-05-08T01:02:09.597 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50802.log.gz 2024-05-08T01:02:09.599 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80781.log 2024-05-08T01:02:09.605 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.52244.log: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45979.log.gz 2024-05-08T01:02:09.606 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39127.log.gz 2024-05-08T01:02:09.616 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79125.log 2024-05-08T01:02:09.616 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.80781.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.80781.log.gz 2024-05-08T01:02:09.623 INFO:teuthology.orchestra.run.smithi064.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-05-08T01:02:09.626 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65146.log 2024-05-08T01:02:09.626 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79125.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79125.log.gz 2024-05-08T01:02:09.641 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71432.log.gz 2024-05-08T01:02:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70334.log 2024-05-08T01:02:09.652 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.65146.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49484.log.gz 2024-05-08T01:02:09.652 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79032.log 2024-05-08T01:02:09.652 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.70334.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70334.log.gz 2024-05-08T01:02:09.654 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51138.log.gz 2024-05-08T01:02:09.658 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47724.log.gz 2024-05-08T01:02:09.663 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56078.log.gz 2024-05-08T01:02:09.664 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66652.log 2024-05-08T01:02:09.664 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79032.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.79032.log.gz 2024-05-08T01:02:09.665 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36911.log 2024-05-08T01:02:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.66652.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66652.log.gz 2024-05-08T01:02:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93758.log 2024-05-08T01:02:09.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.0.36911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45910.log 2024-05-08T01:02:09.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93758.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.93758.log.gz 2024-05-08T01:02:09.671 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41941.log 2024-05-08T01:02:09.674 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62444.log.gz 2024-05-08T01:02:09.678 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.45910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81199.log 2024-05-08T01:02:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.41941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83202.log 2024-05-08T01:02:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.81199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48485.log 2024-05-08T01:02:09.682 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9%/var/log/ceph/cluster1-client.mirror.83202.log: -- replaced with /var/log/ceph/cluster1-client.mirror.81199.log.gz 2024-05-08T01:02:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83991.log 2024-05-08T01:02:09.682 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.48485.log: -- replaced with /var/log/ceph/cluster1-client.mirror.83202.log.gz 2024-05-08T01:02:09.682 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39418.log 2024-05-08T01:02:09.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.83991.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83991.log.gz 2024-05-08T01:02:09.684 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35940.log 2024-05-08T01:02:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39418.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39107.log 2024-05-08T01:02:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70832.log 2024-05-08T01:02:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39107.log: /var/log/ceph/cluster1-client.admin.35940.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35940.log.gz 2024-05-08T01:02:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37761.log 2024-05-08T01:02:09.688 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57832.log.gz 2024-05-08T01:02:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40538.log 2024-05-08T01:02:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.70832.log: /var/log/ceph/cluster1-client.admin.37761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74689.log 2024-05-08T01:02:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.40538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36597.log 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39383.log 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70832.log.gz 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.36597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74390.log 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64904.log 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.74390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36413.log 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74390.log.gz 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.admin.36413.log: 69.8% 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.36495.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.44411.log.gz 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36413.log.gz 2024-05-08T01:02:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.64904.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40445.log.gz 2024-05-08T01:02:09.728 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74689.log.gz 2024-05-08T01:02:09.728 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41941.log.gz 2024-05-08T01:02:09.728 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42313.log.gz 2024-05-08T01:02:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93242.log 2024-05-08T01:02:09.728 INFO:teuthology.orchestra.run.smithi116.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64904.log.gz 2024-05-08T01:02:09.728 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42406.log.gz 2024-05-08T01:02:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.93242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85988.log 2024-05-08T01:02:09.728 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.93242.log.gz 2024-05-08T01:02:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67816.log 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85364.log 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76269.log 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85364.log: /var/log/ceph/cluster2-client.mirror.67816.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85364.log.gz 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85988.log.gz 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46294.log 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.76269.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76269.log.gz 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81622.log 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90341.log 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.81622.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81622.log.gz 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37857.log 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90341.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90341.log.gz 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91869.log 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.37857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50579.log 2024-05-08T01:02:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46666.log 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81696.log 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46666.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91869.log.gz 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85457.log 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.81696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92600.log 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85457.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85457.log.gz 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92797.log 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92600.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92600.log.gz 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87790.log 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.92797.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.92797.log.gz 2024-05-08T01:02:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46759.log 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87790.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.81696.log.gz 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87790.log.gz 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72561.log 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79578.log 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82274.log 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.79578.log: /var/log/ceph/cluster1-client.mirror.72561.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79578.log.gz 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.72561.log.gz 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88030.log 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.82274.log: 88.2% 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.82274.log.gz 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69166.log 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95371.log 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.69166.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69166.log.gz 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54554.log 2024-05-08T01:02:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.admin.95371.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.95371.log.gz 2024-05-08T01:02:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90071.log 2024-05-08T01:02:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.54554.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88030.log.gz 2024-05-08T01:02:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51508.log 2024-05-08T01:02:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90071.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90071.log.gz 2024-05-08T01:02:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91943.log 2024-05-08T01:02:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-05-08T01:02:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91943.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91943.log.gz 2024-05-08T01:02:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55008.log.gz 2024-05-08T01:02:09.733 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.91379.log 2024-05-08T01:02:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79259.log 2024-05-08T01:02:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90589.log 2024-05-08T01:02:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.79259.log: /var/log/ceph/cluster2-client.mirror.91379.log: 83.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91379.log.gz 2024-05-08T01:02:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49898.log 2024-05-08T01:02:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.90589.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90589.log.gz 2024-05-08T01:02:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50536.log 2024-05-08T01:02:09.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.49898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91621.log 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.50536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70787.log 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.91621.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91621.log.gz 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39852.log 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.70787.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70787.log.gz 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91704.log 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.39852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51377.log 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91704.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.91704.log.gz 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87259.log 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51377.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38624.log 2024-05-08T01:02:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.87259.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.87259.log.gz 2024-05-08T01:02:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66288.log 2024-05-08T01:02:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.38624.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57642.log.gz 2024-05-08T01:02:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85016.log 2024-05-08T01:02:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.66288.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91745.log 2024-05-08T01:02:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.85016.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85016.log.gz 2024-05-08T01:02:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66266.log 2024-05-08T01:02:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.91745.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91745.log.gz 2024-05-08T01:02:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50692.log 2024-05-08T01:02:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.66266.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66266.log.gz 2024-05-08T01:02:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88963.log 2024-05-08T01:02:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.50692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48420.log 2024-05-08T01:02:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.88963.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88963.log.gz 2024-05-08T01:02:09.760 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.91446.log.gz 2024-05-08T01:02:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44318.log 2024-05-08T01:02:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.48420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92903.log 2024-05-08T01:02:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.44318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51974.log 2024-05-08T01:02:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.92903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60264.log 2024-05-08T01:02:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.51974.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92903.log.gz 2024-05-08T01:02:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89351.log 2024-05-08T01:02:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.60264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52038.log 2024-05-08T01:02:09.761 INFO:teuthology.orchestra.run.smithi116.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60264.log.gz 2024-05-08T01:02:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46817.log 2024-05-08T01:02:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster1-client.mirror.89351.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89351.log.gz 2024-05-08T01:02:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cluster2-client.mirror.46817.log: /var/log/ceph/cluster2-client.mirror.52038.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65146.log.gz 2024-05-08T01:02:09.784 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47100.log.gz 2024-05-08T01:02:09.789 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47746.log.gz 2024-05-08T01:02:09.808 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47677.log.gz 2024-05-08T01:02:09.813 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42843.log.gz 2024-05-08T01:02:09.828 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44318.log.gz 2024-05-08T01:02:09.828 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39383.log.gz 2024-05-08T01:02:09.830 INFO:teuthology.orchestra.run.smithi116.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36597.log.gz 2024-05-08T01:02:09.831 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39418.log.gz 2024-05-08T01:02:09.834 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39852.log.gz 2024-05-08T01:02:09.834 INFO:teuthology.orchestra.run.smithi116.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52038.log.gz 2024-05-08T01:02:09.835 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40538.log.gz 2024-05-08T01:02:09.835 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37857.log.gz 2024-05-08T01:02:09.836 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49898.log.gz 2024-05-08T01:02:09.840 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79259.log.gz 2024-05-08T01:02:09.842 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44539.log.gz 2024-05-08T01:02:09.876 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.37761.log.gz 2024-05-08T01:02:09.877 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46294.log.gz 2024-05-08T01:02:09.879 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38624.log.gz 2024-05-08T01:02:09.896 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67816.log.gz 2024-05-08T01:02:09.916 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46666.log.gz 2024-05-08T01:02:09.917 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46759.log.gz 2024-05-08T01:02:09.918 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46817.log.gz 2024-05-08T01:02:09.921 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66288.log.gz 2024-05-08T01:02:09.932 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49122.log.gz 2024-05-08T01:02:09.932 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50403.log.gz 2024-05-08T01:02:09.938 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50845.log.gz 2024-05-08T01:02:09.954 INFO:teuthology.orchestra.run.smithi116.stderr: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48122.log.gz 2024-05-08T01:02:09.956 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93591.log.gz 2024-05-08T01:02:09.959 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52244.log.gz 2024-05-08T01:02:09.959 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51310.log.gz 2024-05-08T01:02:09.959 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51377.log.gz 2024-05-08T01:02:09.961 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51508.log.gz 2024-05-08T01:02:09.967 INFO:teuthology.orchestra.run.smithi116.stderr: 94.0% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-05-08T01:02:10.020 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54554.log.gz 2024-05-08T01:02:10.071 INFO:teuthology.orchestra.run.smithi116.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37068.log.gz 2024-05-08T01:02:10.121 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45910.log.gz 2024-05-08T01:02:10.174 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49319.log.gz 2024-05-08T01:02:10.299 INFO:teuthology.orchestra.run.smithi116.stderr: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37653.log.gz 2024-05-08T01:02:10.302 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50692.log.gz 2024-05-08T01:02:10.374 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48420.log.gz 2024-05-08T01:02:10.378 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39107.log.gz 2024-05-08T01:02:10.379 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48485.log.gz 2024-05-08T01:02:10.502 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50536.log.gz 2024-05-08T01:02:10.599 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51049.log.gz 2024-05-08T01:02:10.989 INFO:teuthology.orchestra.run.smithi116.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50579.log.gz 2024-05-08T01:02:12.566 INFO:teuthology.orchestra.run.smithi116.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.37007.log.gz 2024-05-08T01:02:14.730 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39315.log.gz 2024-05-08T01:02:14.734 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37451.log.gz 2024-05-08T01:02:14.876 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37789.log.gz 2024-05-08T01:02:15.151 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46094.log.gz 2024-05-08T01:02:17.028 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47769.log.gz 2024-05-08T01:02:19.247 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47859.log.gz 2024-05-08T01:02:19.642 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43037.log.gz 2024-05-08T01:02:21.352 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52589.log.gz 2024-05-08T01:02:21.430 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52381.log.gz 2024-05-08T01:02:23.242 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49234.log.gz 2024-05-08T01:02:24.320 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48236.log.gz 2024-05-08T01:02:25.217 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51895.log.gz 2024-05-08T01:02:25.497 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48602.log.gz 2024-05-08T01:02:25.811 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52499.log.gz 2024-05-08T01:02:26.306 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52159.log.gz 2024-05-08T01:02:26.872 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51974.log.gz 2024-05-08T01:02:27.937 INFO:teuthology.orchestra.run.smithi116.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51752.log.gz 2024-05-08T01:02:28.397 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52061.log.gz 2024-05-08T01:02:29.162 INFO:teuthology.orchestra.run.smithi116.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52723.log.gz 2024-05-08T01:02:29.353 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-05-08T01:02:29.913 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48145.log.gz 2024-05-08T01:02:29.964 INFO:teuthology.orchestra.run.smithi116.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37676.log.gz 2024-05-08T01:02:30.153 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38900.log.gz 2024-05-08T01:02:30.494 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52267.log.gz 2024-05-08T01:02:31.252 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51684.log.gz 2024-05-08T01:02:32.433 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52869.log.gz 2024-05-08T01:02:32.897 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37352.log.gz 2024-05-08T01:02:33.886 INFO:teuthology.orchestra.run.smithi116.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38793.log.gz 2024-05-08T01:02:34.670 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48508.log.gz 2024-05-08T01:02:34.686 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51814.log.gz 2024-05-08T01:02:34.836 INFO:teuthology.orchestra.run.smithi116.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46002.log.gz 2024-05-08T01:02:36.197 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39172.log.gz 2024-05-08T01:02:37.128 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42931.log.gz 2024-05-08T01:02:38.571 INFO:teuthology.orchestra.run.smithi116.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36939.log.gz 2024-05-08T01:03:03.686 INFO:teuthology.orchestra.run.smithi116.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.36939.log.gz 2024-05-08T01:03:13.067 INFO:teuthology.orchestra.run.smithi116.stderr: 94.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36911.log.gz 2024-05-08T01:03:57.905 INFO:teuthology.orchestra.run.smithi064.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-05-08T01:04:06.764 INFO:teuthology.orchestra.run.smithi116.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36911.log.gz 2024-05-08T01:04:08.712 INFO:teuthology.orchestra.run.smithi064.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-05-08T01:04:21.071 INFO:teuthology.orchestra.run.smithi064.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-05-08T01:04:21.073 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-05-08T01:04:21.073 INFO:teuthology.orchestra.run.smithi064.stderr:real 2m14.405s 2024-05-08T01:04:21.073 INFO:teuthology.orchestra.run.smithi064.stderr:user 3m22.783s 2024-05-08T01:04:21.073 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m7.637s 2024-05-08T01:08:09.518 INFO:teuthology.orchestra.run.smithi116.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-05-08T01:08:34.069 INFO:teuthology.orchestra.run.smithi116.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-05-08T01:09:10.961 INFO:teuthology.orchestra.run.smithi116.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-05-08T01:09:10.962 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-05-08T01:09:10.963 INFO:teuthology.orchestra.run.smithi116.stderr:real 7m4.263s 2024-05-08T01:09:10.963 INFO:teuthology.orchestra.run.smithi116.stderr:user 5m9.591s 2024-05-08T01:09:10.963 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m15.819s 2024-05-08T01:09:10.963 INFO:tasks.ceph:Archiving logs... 2024-05-08T01:09:10.964 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630/remote/smithi064/log 2024-05-08T01:09:10.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-08T01:09:40.145 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630/remote/smithi116/log 2024-05-08T01:09:40.146 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-08T01:10:25.489 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-08T01:10:25.502 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-08T01:10:25.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-08T01:10:25.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-08T01:10:25.598 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-08T01:10:25.608 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-08T01:10:25.609 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T01:10:25.612 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T01:10:25.628 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-05-08T01:10:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-05-08T01:10:25.658 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T01:10:25.658 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-05-08T01:10:25.658 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 57 +96us[ +111us] +/- 72ms 2024-05-08T01:10:25.658 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 +536us[ +551us] +/- 65ms 2024-05-08T01:10:25.658 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 -771us[ -755us] +/- 78ms 2024-05-08T01:10:25.658 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T01:10:25.669 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-08T01:10:25.670 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-05-08T01:10:25.670 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 54 +166us[ +166us] +/- 72ms 2024-05-08T01:10:25.670 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 +598us[ +630us] +/- 65ms 2024-05-08T01:10:25.670 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -708us[ -675us] +/- 78ms 2024-05-08T01:10:25.670 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-08T01:10:25.670 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-08T01:10:25.680 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-08T01:10:25.680 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-08T01:10:25.708 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-08T01:10:25.799 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-08T01:10:25.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-08T01:10:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T01:10:25.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-08T01:10:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T01:10:25.994 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-08T01:10:26.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-08T01:10:26.056 INFO:teuthology.task.internal:Duration was 2894.329790 seconds 2024-05-08T01:10:26.056 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-08T01:10:26.093 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-08T01:10:26.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-08T01:10:26.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-08T01:10:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T01:10:26.179 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-08T01:10:26.655 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-08T01:10:26.655 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-05-08T01:10:26.655 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-08T01:10:26.681 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-05-08T01:10:26.682 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-08T01:10:26.702 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-08T01:10:26.702 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T01:10:26.724 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T01:10:26.769 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-08T01:10:26.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-08T01:10:26.796 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-08T01:10:26.952 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-08T01:10:27.224 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-08T01:10:27.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-08T01:10:27.253 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-08T01:10:27.308 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-08T01:10:27.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-08T01:10:27.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-08T01:10:27.431 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-05-08T01:10:27.456 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-05-08T01:10:27.477 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-08T01:10:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T01:10:27.534 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-08T01:10:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T01:10:27.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-08T01:10:27.560 INFO:teuthology.task.internal:Transferring archived files... 2024-05-08T01:10:27.561 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630/remote/smithi064 2024-05-08T01:10:27.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-08T01:10:27.609 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-05-07_23:02:08-rbd-wip-dis-testing-squid-distro-default-smithi/7696630/remote/smithi116 2024-05-08T01:10:27.609 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-08T01:10:27.643 INFO:teuthology.task.internal:Removing archive directory... 2024-05-08T01:10:27.643 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-08T01:10:27.646 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-08T01:10:27.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-08T01:10:27.710 INFO:teuthology.task.internal:Not uploading archives. 2024-05-08T01:10:27.710 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-08T01:10:27.735 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-08T01:10:27.735 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-08T01:10:27.738 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-08T01:10:27.751 INFO:teuthology.orchestra.run.smithi064.stdout: 265585 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 8 01:10 /home/ubuntu/cephtest 2024-05-08T01:10:27.753 INFO:teuthology.orchestra.run.smithi116.stdout: 265307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 8 01:10 /home/ubuntu/cephtest 2024-05-08T01:10:27.755 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-08T01:10:27.776 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-08T01:10:27.844 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} duration: 2894.3297896385193 flavor: default owner: scheduled_dis@teuthology success: true 2024-05-08T01:10:27.844 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T01:10:27.935 INFO:teuthology.run:pass